Hacker News Reader

Comment by korginator | Hacker News Reader

Kudos to BB for their swift actions.

Under no situation should any software randomly delete folders and files it doesn't own, period, specially something so widely used by professionals. This is not just a little bug on a rarely used OS platform not limited to people using backblaze on the Mac, it should have been a show-stopper, and should have never left the test environment (do they even have one?). It reflects on the abysmal quality of Adobe's software and their QA.

There's no reason Adobe should even touch the root directory "/" on the Mac, leave alone delete anything from there. Why does Adobe CC need root permissions to install on the Mac in the first place?

I hope the people responsible for this disaster, their managers and the QA team were fired. In my business, this sort of mess can sink our company.

View on HN

> I hope the people responsible for this disaster, their managers and the QA team were fired. In my business, this sort of mess can sink our company.

I hope something very different: I hope they're required to write a thorough post-mortem analysis of what went wrong, and what in their development & testing process led to the failure. And then required to implement improvements to both that would prevent this class of bugs from happening again.

Firing people for a mistake is rarely a good idea. It makes them not want to admit mistakes. Firing them for making the same mistake several times and not learning for it -- fire away.

c.f., http://itrevolution.com/uncovering-the-devops-improvement-pr... and the "blameless post-mortem"

Yev from Backblaze here -> That would be cool, though there's no real "requiring body" to require post-mortems. The company has to want them. We do them ALL the time internally at Backblaze, they help us piece together what the hell went wrong and gives us next steps to work on. We find it really useful, especially with software/hardware issues (less so UI glitches and stuff).

When you find yourself writing code that deletes directories, you need to be extraordinarily careful. It should be subject not only to extensive testing, but a design review that questions if there isn't a better way to achieve your goal because of the many ways code that runs rm -rf can go wrong.

If code like this escapes code review and testing and wacks actual customer data, firing is an appropriate response.

In the code I've written that deletes directories, I never call the system level delete functions directly, and instead have abstracted them away to a function with added checks to the beginning that fail/assert with big warning messages if parameters ever try to delete root or other system directories.

It's never been triggered except by my unit tests, but I'm sure glad it's there.

Are you sure, without seeing the bug? Imagine two different manifestations of the bug.

(a): Pull out first directory found in readdir("/"), system("rm -rf ${dirname}") ( in no real pseudocode).

(b) removeCacheDirs(); --> recursiveDelete(getLocalCacheDirList()); --> ReadConfig(GetUserPrefLocation(CACHE_DIR)), .... <some horribly nested mess of functionality created by 10 different programmers over as many years, where somehow the code misbehaves under a specific set of environment attributes, etc.>

Sure - (a) would be awesomely stupid, but it's probably not what happened. Oh, it might be. But (b) is more likely, and reflects as much an organizational and dev process mess as anything. And if (a) happened, you need to ask yourself how: Why was someone with so little clue allowed to implement something like this with no code review / why did it bypass code review? What's the longer-term policy and mechanism to prevent this?

When a three year old shoots someone with a gun, do you put the three year old in jail, or do you figure out how the $*! they got a gun in the first place?

When I wrote similar functionality, we had (as you mentioned) various helper functions. All of them where heavily tested and marked for extra code review if they were changed. And if you need a temp or cache dir, systems provide safe ways to create such -- eg mktemp, GetTempPath, etc.

And it's more than fair for the lead who approved the code change, and perhaps even the lead who approved the entire design to have their jobs on the line for this.

  It reflects on the abysmal quality of Adobe's software
  and their QA. [...] I hope the people responsible for
  this disaster, their managers and the QA team were fired.

Eh, I can kind of see how it might happen myself. Remember that time Valve had a bug in Steam for Linux that would wipe the user's hard drive? [1]

It's easy to imagine how this sort of thing would happen - manual test guys don't have hidden folders in their root directories because their boxes are locked down by corporate IT and they can't create any folders in the root directory. Developers don't have them because they understand the unix way and know they shouldn't be putting files there. CI servers don't have such folders because, well, why would they? None of the test environments include the specific condition needed to trigger this bug - a hidden folder in the root directory - so the bug never appears in testing.

Is it bad and serious? Absolutely. But I don't think the fact testing missed this is a sign of testing incompetence - you could have cutting edge testing following every industry best practice and still miss this bug.

[1] https://github.com/ValveSoftware/steam-for-linux/issues/3671

The Steam bug was the first thing that popped into my mind...But does application testing have to be so dependent on testers having all possible permutations of a system configuration? I mean, I can't offer a mathematical proof for this, but it seems to me possible to write code that pre-empts the situation where an arbitrary directory is blown away, willy-nilly. In the same way that while it's hard to effectively blacklist all the dangers, it's still possible to effectively implement security by whitelisting allowed behaviors.

For the Steam bug, it could have been fixed by running the shell script with options that prevent use of undefined variables.

It really isn't hard to have a company policy about nounset, errexit, and, in the extreme, noclobber.

Don't forget companies with a nolock policy.

What's nolock?

http://blogs.msdn.com/b/davidlean/archive/2009/04/06/sql-ser...

That's exactly the point of application sand boxing, which Apple's been pushing for a while.

Honestly? Adobe has been pulling this sort of bad stuff for a while now, it is not just a dumb mistake.

I am a hardcore fan of Macromedia Fireworks, and Adobe releases caused so much problems that I decided to use Macromedia Fireworks in the literal sense (ie: in my current machine, I have Macromedia Fireworks 8 installed).

When Adobe bought Macromedia, I installed their Adobe Fireworks thinking it would be just an updated Fireworks... it was right, and wrong, right because it barely had any new features or bug fixes, wrong because it install for some reason needed "adobe common files" anyway, and installed 3gb of crap on my disc (when discs were still 20gb big, thus it was a huge waste of space), AND somehow it conflicted with their own Adobe Reader and made it stop working.

When Adobe Fireworks CS3 was released, I was in university, I was excited to try it... and not only found again it had almost no new features, old features don't worked correctly either, most aggravatingly, the selection boxes would always render in the wrong place, and not just 1 pixel off, they would be completely off, when I tried to input text for example, the text box appeared 300 pixels away from the actual text.

When a friend installed it on his computer, the computer started misbehaving.

And the problems kept coming, to the point that eventually I gave up, I have no idea what version Fireworks is now, because I decided when CS6 came out, to install on my machine Macromedia Fireworks, and not install any Adobe product on my computer (eventually I found it was hard to avoid Acrobat Reader, thus now I install it, but only that).

EDIT: Also I always disable Adobe Updater, the first thing I do after installing any Adobe product on any computer for any reason, is shut down the updater process, and forbid it from running.

can relate to that. adobe's arrogant shit show destroyed the best fireworks.

Adobe EOLd Fireworks in 2012.

I go one step further - I block all access to Adobe's update servers at the router so it can never get there.

Old Macromedia Fireworks (and current Adobe Cloud) user here. Fireworks still has not changed in any positive way since Macromedia developed it.

The bright side is that they haven't completely ruined it. I haven't noticed the text box problem, but I skipped many versions before I switched over to OSX and had to get the new one.

Except that it's not just a hidden folder in the root directory. It's the first folder sorted by name in the root directory...

Which if you have no hidden files .in / ends up being /Applications !

> Remember that time Valve had a bug in Steam for Linux that would wipe the user's hard drive? [1]

Counter-point:

* Let's be honest, Linux is not a first class gaming platform. Linux is not going to be Steam's #1 platform. Whereas here it's an Adobe product on OSX.

* The Steam bug was triggered by the user moving ~/.local/share/steam. That's not something that users will be doing regularly. Whereas this Adobe bug wasn't caused by any uncommon behaviour.

> Developers don't have them because they understand the unix way and know they shouldn't be putting files there.

Developers are very definitly going to have dot-files. .ssh, .vim, .bashrc, .local, .gnome, .mozilla, etc. etc. If something wiped my SSH private keys on my desktop, that would be a massive problem.

  Developers are very definitly going to have dot-files. .ssh

The article only mentions the system root directory, never the user's home folder. If you're storing your ssh keys in your system root directory, you're doing something very unusual.

Perhaps a bit pedantic (HN!) ... We may be remembering a different bug but wasn't the steam bug a failure to check that a variable existed before using it to complete a directory that was to be deleted.

The variable was empty when a user moved steamroot but that would have been fine with proper input checking - also there were probably other scenarios where that failure soul result in catastrophe too.

http://m.theregister.co.uk/2015/01/17/scary_code_of_the_week...

I'll be running chattr +i on my private keys when I get a chance. I wonder why that bit isn't set by default?

> It's easy to imagine how this sort of thing would happen - manual test guys don't have hidden folders in their root directories because their boxes are locked down by corporate IT and they can't create any folders in the root directory.

This is what QA is for. It should be thorough, more thorough than you imagine and performed by someone other than who wrote the code.

> Eh, I can kind of see how it might happen myself

Exactly! Any programmer worth their salt should be able to see how it might happen... and prevent it!

> you could have cutting edge testing following every industry best practice and still miss this bug

I disagree. If you are following industry best practice (checking files/directories before you delete them) you would not have this bug.

Adobe is the company who once thought modifying the boot sector of a hard drive was OK [1].

I'm just glad they didn't migrate to messing around with EFI variables for their copy protection.

1: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/441941/...

FWIW FlexNet is not an Adobe product, it's by Flexera[0] (Acresso at the time, prior owner Macrovision). It is/was used by a lot of products for license enforcement. I wasn't personally aware of the boot sector crap until now though.

Doesn't really excuse Adobe.

[0] https://en.wikipedia.org/wiki/FlexNet_Publisher

I've implemented FlexNet licensing in some of our software and my first thought when I read about this adobe bug was that it was an error in the FlexNet licensing implementation since it likes to just own a folder and wipe its contents before refreshing licensing info from the server (which would make sense after an install/update).

The only thing I could find from Adobe was the basic release notes, which really don't say much. Has anyone seen more details on how this happened? If it is licensing, I doubt they'd admit it.

Messing around in /? Pretty certain it is licensing, putting information in obscure places most users won't look for them.

> I hope the people responsible for this disaster, their managers and the QA team were fired.

This is a actually a rather typical response, made with full awareness of the consequences of an action by people not under the same constraints as the participants in an incident. Diane Vaughan (an expert on risk, causes of incidents, and how organizations come to become error-prone) terms these actions 'ritualistic':

> Causes must be identified, and in order to move forward, the organization must appear to have resolved the problem. The direct benefit of identifying the cause of an accident as Operator Error is that the individual operator is the target of change. The responsible party can be transferred, demoted, retrained, or fired, all of which obscure flaws in the organization that may have contributed to the mistakes made by the individual who is making judgments about risk. The dark side of this ritualistic practice is that organizational and institutional sources of risk and error are not systematically subject either to investigation or other technologies of control.

To make it even better, you're now throwing away all the people most experienced in the conditions that lead to a specific incident. Unless very specific conditions are true, this is a sub-optimal recommendation. The evidence seems to be against building a culture of fear.

There's no reason Adobe should even touch the root directory "/" on the Mac,

There is no reason why an application should touch anything outside its own sandbox (unless requested by a user via a open/save dialog). Incidents like this show why developers should distribute their Mac applications sandboxed by default.

> I hope the people responsible for this disaster, their managers and the QA team were fired.

Yes, call out the lynch mob.

Coincidentally, this bug was released a day before a holiday weekend. It's practically guaranteed that people were crunched on their deadlines and faced with either a) working all weekend b) getting reprimanded for failing to meet their deadlines or c) calling it good, releasing with inadequate review, and enjoying the long weekend (or so they thought). Firing minions and writing technical post-mortems won't help with creating a sane work environment.

I saw something interesting last week where an upgraded osx to el cap screwed up the adobe fodder permissions lib. Adobe's suggested fix was to give full access to everyone. I don't suggest upgrading to el capitan, clean install only. I also suggest people use standard accounts, and authenticate updates and installs with a separate admin account. I'm eager to see what the real reasoning behind the bug was, interesting programmer error using osx API I imagine.

It sounds like it touched the homedir, not the root dir.

Is there any kind of docker-like container solution for macosx? In a docker-like world, this wouldn't happen would it? An installed application (in a container) wouldn't have access to delete in a willy–nilly manner.

Reply on news.ycombinator.com