Fired employee wipes out whole season of TV show
msnbc.msn.com
msnbc.msn.com
You mean like a pinata?
Actually, outsourcing is designed to be cheap, not to lower risk. Trusting the lowest bidder to an important task usually results in disappointment.
If all the legal work was done properly, the vendor would end up in court with their insurer.
I think maybe the TV producer's lawyers think that suing the internet host into oblivion will get their TV show back?
If this was indeed a risk transfer outsourcing agreement, everyone who was involved in vetting and approving the hosting contract should be fired and possibly sued for negligence.
There's up to two fools here:
- If the hosting company does not have a disclaimer in their contract that the data backup onus is not on them, then they deserve to be sued for all they got.
- The TV producer should keep their own backup. Trusting a third party blindly with your most precious data is simply stupid, and unworthy of anyone involved with IT in this day and age. I would back up data even if the host did have a contractual obligation to back up my data. Shit happens, it's not a matter of "if", but of "when".
Here's the company's web page detailing their data backup plan: http://www.cyberlynk.net/services_corporate/dsp_backup.cfm
EDIT: Case in point: We probably all agree that dropbox could be called "data storage", but they specifically say that backups and backup costs are to be done by the client (http://www.dropbox.com/dmca#terms), not by them.
Yes, your vendor may have their own redundant systems in place so that their internal problems don't become your problems. And that's nice. They should. But you always have a Plan B. And you always keep a local copy.
Surely the producers have at least working copies somewhere (I can't see people live-editing on a cloud server). Surely they considered the risk of the company going bust / data loss / their own staff deleting the copies.
Would assume this was an April Fools thing it's that bad (except for the fact it quite obviously isn't due to the quite obvious reputation damage that will effectively destroy this company).
The data breach allegedly knocked out 6,480 WER1 electronic files, or 300 gigabytes of data, comprising two years of work from hundreds of contributors globally, including animation artwork and live action video production."
These sort of reports often over do it with the use of "allegedly" as an attempt to mitigate any libel claim.
"Jewson is alleged to have been charged"
Surely it's a matter of record whether he was charged. If the reporter can't even confirm this then they don't have a story.
"The data breach allegedly knocked out 6,480 WER1 electronic files"
Again they don't need that "allegedly", it's presumably a reported fact, the allegation they have to be wary off is the one that says the person responsible.
This really annoys me, allegedly.
While I agree with you on your first point (that the reporter shouldn't have to use the word "allegedly" if he/she can confirm the man was charged), personally, I think using allegedly in the sentence about the files being knocked out is warranted. "The data breach" is referring to the data breach the man *allegedly carried out.
TV news reporters != attorneys. So when in doubt, attribution is your best friend.
Thanks for your viewpoint though.
Sounds like they hosted everything in the "cloud". Consider what happens when you edit a Google doc, there is no offline backup just whatever Google provides. So sure they are not at zero, but plenty of stuff can and was lost.
And keeping everything in the cloud would be insane. In our shop every video editing workstation had it's own RAID array just for realtime editing. In a large shop you'd have a SAN of some kind.
I have heard stories of letting people go to find out that all of the work that employee did for the last month or two deleted, source missing, machines just wiped clean.
I read that title and kind of laughed because, well, it is a seemingly commonplace response in some parts of the world.
This was the hole in the policy. It was designed to keep disgruntled employees from damaging the system but the ones leaving on good terms were treated exactly the same.
I have worked on financial software the majority of my career; at one point in a Wall St. investment banking firm. The very explicit protocol there was if you were going to leave the 2 week notice was customary, but you would probably be locked out of everything instantly, and would be around for hands-off consulting at the whim of your manager for the duration. It was all handled very professionally and friendly, at least for me, and they let me keep working for the 2 weeks since anything I touched had a LOT of layers to go through before it ever saw production data.
It wasn't all their data but enough to make what was left unusable.
Because there are many terms for that particular personality type, none of them flattering.
(In the end, putting someone in a situation where they'll do something bad is worse than being the person that does something bad. A good example is the Colgan Air crash from a few years ago. Anybody knows that when your plane is stalling you're supposed to push the control column forward and apply full engine power. The plane does the pushing the control column forward part automatically! But for some reason, the trained airline captain did the exact opposite, stalling the plane and killing everyone on board. Why? Because he got paid so little that he couldn't afford a hotel room to sleep in before his work that day, and had to commute across the country and then work a full shift. This is the airline's fault for impairing someone to the extent that he killed 50+ people by doing the exact opposite of what any trained pilot would do. $300 for a hotel room and everyone on that flight would still be alive today.
Similarly, when you fire someone that works with critical data, you need to make sure the guy's access is revoked. It's a precaution that needs to be taken, as this case shows. When necessary precautions are omitted, bad things happen. Planes crash, and important data goes away forever.)
Upvoted for exactitude. Also, it's worth remembering that virtually no one hires already disgruntled employees. That's not to excuse anything that the truly disgruntled do to retaliate. But it does suggest that an abusive employer represents a risk to everyone relying on their 'teams'.
EDIT: I don't think the pilot was retaliating, by the way. What he was subject to is a different kind of abuse.
Separately, when two parties are in a fight, it's always worth remembering who started it. If blame needs to be shared, and it's unfair to share it evenly, the unprovoked aggressor deserves a special measure of approbation. Typically, this is the party with more power, not less, for the simple reason that it's a lot harder to abuse a lack of trust and authority.
Except companies that believe this are impossible to work for, there is so much process and policy to prevent anyone doing any harm that no-one can get any work done either. At the end of the day, you have to sort out all the trust issues before you give someone the root password, then you just have to trust them to get on with it.
It was someone's job to avoid this from happening, or they do not have people covering such cases... either way, someone other (person or the company as a whole) than the fired employee messed up for the business/customers and they are liable/answerable.
Because people fail... all the time. Good, Smart, Sincere people fail. (not in this particular case, though)
This is just how things are. We all have heard numerous stories of rm -r / or drop database or something similar. Hell, the whole widely accept idea of "bugs" in software industry is based on the fact that people WILL do something wrong. Not because they are bad people, want to do bad things, but to err is human.
So your process should not be designed around the idea of people doing the right things always.
This is what I think he means (and I concur)
I mean this completely academically, so please don't take it as an insult (I'm genuinely curious) - but was this perspective something you were raised with or did this develop out of experience / general personal exploration of the world?
Having intent to destroy something on purpose is a different matter though, and I'm not sure that there is a good argument for it.
There is a sort of half intent by the developer. Yes, the developer had intended to delete the repository that was in his home directory, but he had no idea that it was actually a link to the master repository.
A symlinked directory, though, being made victim of a recursive rm command, just might result in rm traversing through the directory and deleting everything inside of it.
As a producer, I can safely say that your entire job is based around one rule: the show must go on. That means you have backups. And your backups have backups. And your backup's backup's have backups.
Is it your fault when bad things happen? If you're good at what you do, probably not - especially when you consider the AMAZING number of things that can go wrong on any given day. However, it IS your fault if the bad things that inevitably happen cause the show to fail because you had no plan B.
Maybe it's a disgruntled employee with a set of passwords he really shouldn't have. Or maybe it's an outbreak of anthrax (yeah, I was on a job where that was suddenly an issue). Perhaps you're shooting on a beach and a dead body randomly washes up, suddenly turning your carefully chosen location into a crime scene crawling with cops and coroners (another true story / very rough day.) God forbid it's something really awful, like a bike messenger carrying an important batch of media getting hit by some asshole who was answering his phone instead of paying attention to the road, landing the kid in hospital, and turning your package into collateral damage (horrific moments like this, by the way, make the job seem suddenly unimportant).
Regardless, the show must go on. And if you have a good producer, it will. But losing TWO YEARS worth of work because one data center (effectively) went offline? That's seriously embarrassing. Also, very bad for credibility. And yes, reason enough to loose your stripes.
That's why every CEO is responsible for everything that goes on in their company, whether or not they knew every little thing that was going on. It's their responsibility to know, and put processes in place to ensure that they know, and if something still slips through the cracks, own up and take action to fix (or take the fall if it's a big enough issue).
What I think people are missing is that this doesn't absolve the perpetrator of moral, legal, and professional responsibility for a clearly malicious act. So two parties are at fault and should experience the pain in different ways. Fact remains, they should both experience pain.
However, that wasn't the assertion that I was intending to respond to :) I've seen this "its not the perpetrator's fault" argument used several times in the last few weeks, the most recent that I remember was the kid who hacked the PHPFog site. The same argument was used there, and indeed was spouted off by the kid himself - it wasn't the hackers fault the site got hacked.
This line of reasoning seems dangerous to me, as it obscures a criminal or unethical activity by the ultimate result of that activity. Wrong action is wrong action, regardless of who didn't cover their bases.
Should the producer take more precautions, and will they ultimately be burned? Most assuredely, but lets not forget the reason they were burned in the first place - somebody maliciously acted against them.
Rather, I think the issue is the lack of a good backup strategy. Off-site, automated backups should have been in place. I've seen someone `rm -rf /var/mysql` and, after a heart attack, recover it. Your system should be rm-proof!
1. Don't use per-machine unix accounts. Use something like NIS or LDAP. Now you can easily disable someone's account on all boxes. THe bonus is that their home directories still stick around (though you can't use aliases like ~username to access them).
2. Use an email system where you can disable a user without deleting all of their emails.
This is wrong; and is a class of error which I see frequently, especially in political comment; both from the left and the right.
The underlying incorrect assumption is that responsibility is mutually exclusive.
In many cases, responsibilities are divided cleanly between people, but this is an organisational convenience, not a moral law. There are cases where it is inescapable that more than one person has the same responsibility. Consider an army sargeant: he may assign a task to a soldier, but still has the responsibility for seeing that it is done. This is true for all leadership positions.
Much political discourse revolves around who has responsibility, and frequently one sees it argued that A is not responsible because B is, often in cases where this is incorrect. Eg, employer versus employee, individual versus government, etc etc.
The error is made in both directions, depending on the political beliefs of the arguer: an individual is not responsibly because the organisation is, or the organisation is not responsible because the individual is..
A particularly dumb one, in my opinion, is the argument that government is not responsible for something, because individuals are. In my view, all the valid responsibilities of government are derived from responsibilities of the individual: the responsibility to defend the nation is derived from the individuals responsibility to protect him/herself and family, etc.
Examples I'm less sure about: Who "owns" something, whether something is "clean" or "dirty", whether two things are the "same", whether something is "normal" or not. I collect these; if anyone has more to share I'd love to hear them.
Personally I'm convinced it is one of the cognitive flaws that are needed to stop us from going teeth-gnashingly insane after ten minutes exposure to reality.
https://secure.wikimedia.org/wikipedia/en/wiki/Fundamental_a...
See also: "She should have known what would happen to her, going out at night dressed like that."
No.
I would assume it's more about people assuming something like this would never happen as to not justify the outlay of protecting against it.