A Dropbox account gave me stomach ulcers | Hacker News Reader

A Dropbox account gave me stomach ulcers

reddit.com

348 pointsslyall122 comments

I am not a fan of Dropbox. Stories like this are why I don’t use it.

A couple of years ago I inherited managing Dropbox for a company I was at. I’d never really used Dropbox in a more daily use company.

I discovered Dropbox wasn’t very IT admin friendly. If you share out a personal folder as your team folder, IT can’t find your folder to share it to new members of your team. The person who owns the folder has to share it out. So when new employees ask why IT can’t access and share the folder, we had to explain that’s how Dropbox works.

The other crappy thing I discovered about Dropbox from that time. I have no way to audit or tell who has what folder. If someone was sharing something illegal from their folder, I had to turn on the auditing capability and download logs as they were generated. It was confusing because if someone were sharing something and I was the admin of that account, I should have access to all files and accounts. Nope. So we dumped Dropbox before I left to avoid scrutiny down the road for any acquisitions of the company.

Dropbox is still a consumer product trying to be a workplace tool IMO. Maybe others have a different opinion.

This is a good story to tell on Monday.

> If you share out a personal folder as your team folder, IT can’t find your folder to share it to new members of your team. The person who owns the folder has to share it out.

> It was confusing because if someone were sharing something and I was the admin of that account, I should have access to all files and accounts. Nope.

In both cases, you should be able to assume the shared folder owner (via the admin console) and modify their permissions. Did this not work for you?

(note: I'm an ex-Dropbox eng that worked on a lot of the sharing/team management systems)

> The other crappy thing I discovered about Dropbox from that time. I have no way to audit or tell who has what folder.

Have felt similar pain with G Suite Drive. Perhaps I simply didn’t find or didn’t have the proper permissions (I wasn’t the top level admin on the org) but at some point I wrote a Python script to output a list of users who had access to each file in our organization. I saw no way of generating this automatically.

Drive can be a nightmare.

what do you suggest as an alternative. asking for a friend

What I find interesting is that the “insane asylum” apparently knew that it was wrong.

e.g. when asked why they need so much space on their laptops, they were evasive.

To my mind this makes it much worse. Not only were they doing something dumb, but they knew it was dumb, and they hid it instead of fixing it.

It certainly invites speculation about the company culture in the acquired business.

Based on my experience: IT was well aware it was insane. Alas, IT got absolutely no budget for fixing it from top brass, as kicking the can down the road always made more sense short-term

IT of the insane company, or the acquiring company? Because the post says IT of the acquiring company didn't know until the problem blew up.

DD failed to find it during the M&A, issue didn't become apparent until the purchased company lost multiple employees who kept the plates spinning.

It's a failure all round but i would say acquiring companies IT team are least to blame here.

From the article, it doesn't seem to be the fault of the IT guys. It's more like some rich guy who goes out buys a shiny used plane and hands over to the pilot to use.

According to OP, the plate-spinning employees were 'let go' at the time of the acquisition - presumably a decision by the acquiring company.

I worked for a small company right out of college and we had some monsters under the bed too. Unfortunately the manager would rather new code written than cleaning up technical debt or resolving risks. So I would have to double my estimates and use my “spare time” to try to work toward some level of sanity... but there were still plenty of things that I felt guilty about and would be evasive about if someone asked.

It is better not to take ownership of someone else's bad decisions. At the very least, make sure that you do not become complicit in something unethical or illegal.

It also certainly invites speculation about the due diligence of the acquiring company.

I worked in a software company once where we acquired a product to integrate into ours. It was developed by a self taught programmer in his 50ies that did versioning by making copies of the whole directory when he had something that worked, the way I also worked before university. Since he was the only developer and he was pretty good this mostly worked. His first name was/is Leif so from then on me and my friends speak of this style of "version control" as "using Leif Points". I guess because it is phonetically somewhat close to Save Points like in some computer games.

- Are we using git or Leif Points for this project?

- eh, how about git?

This story sounds like they also used Leif Points, but for their database :O

I won't name the offender, but a well-known company I worked for did something funnier: zipping entire directory and checking the zip files in Visual Source Safe.

Given how broken VSS is, that might have been safer than using VSS as if it worked properly.

LOL, I used it in 1996-97 and it worked OK for a modest setup with only six users and blocking at file level. Later heard some horror stories from people working at big shops, but no worse than svn stories, anyway I haven't seen any of those.

About your time of using VSS, I was working at Microsoft in developer support (working on COM data access, and not VSS, thank $DEITY). It was fine for small internal stuff we’d do, the stories I heard from industrial-strength use did not instill confidence for anything more than a half-dozen devs. OTOH, we shipped small components for a larger product using VSS, and I don’t recal any issues. Hated the workflow and interface, but it worked

TBF making tarballs and storing that was how many projects used to work. Linus did that until they got Bitkeeper & git, he refused to use any existing VCS before that.

To be fair VCS before that sucked pretty badly.

I use TFS for source control, but still also use Leif Points (like scheduled backup of the project every day to a local storage) in case of something messed up by the server. So don't underestimate the old guy :)

Why would you call out someone like that?

> 5. Audit has some explaining todo.

Yes, that's a total fuck up on their side as well. That an incompetent company dies while doing stupid stunts like that is understandable and how the market filters out stupidity.

That another company looks at it and thinks "Oh that looks fine", that's a total failure at their work.

What "audit" are we talking about here, though?

Usually when you acquire a company you sign of some intent lettet that shows you are serious about the purchase provided they can pass your smell test.

Then after NDA's and a few other docs if a company accepts being purchased they have to give access to their internal docs. Finance books, IT, etc.

If during this "due diligence" period the buyers find a major red flag they might pull out, revise the price down, etc.

Having seen a couple of acquisitions what I see is that things go forward because the top execs are close and went to the same school and so on. The technical details shared are light and often don't matter until things are settled and the plebs have to merge everything.

It sounds like he's referring to the due diligence as part of the M&A.

Personally I've never seen one of these discovery phases bring back anything remotely close to real life. No random IT manager alive is going to throw his dirty laundry out to a potential buyer and potentially torpedo a deal.

Some of the things I've seen/found during my company buying spree since 2015 have been magical.

I actually have had the reverse experience.

My company was acquired 18 months ago. We were the smaller company, about 1/4 the size, and as the founder/CTO, I expected the acquiring company to be all over us. I mean, we had a pretty tight operation but there was only so much 16 people could achieve. There were plenty of things we could improve, and I was looking forward to the resources and experience of this bigger company to help us.

Was I in for a shock!

While their software dev practices were OK (certainly no better than ours), what hit me hardest was their secops. It was a nightmare. And these guys service some brand name clients.

As an example, not only was the corporate wifi password memorable - not actually “P455w0rd” but similarly bad - but it was written on a whiteboard in view of the whole office! Service passwords are still stored in clear text in the corporate wiki, and few people understand why this is terrible. It’s like a 90s IT shop, where people used to ask me “why would anyone care about our password?”

Kicking the can down the road is fine for a while, but without strong technical leadership, kicking the can becomes how things get done, and that’s been my experience here.

I’m not long for this place.

> Service passwords are still stored in clear text in the corporate wiki.

I know this painfully well :(. Can you recommend some good solution? Ideally open source and self hosted, instead of cloud password manager.

https://github.com/gopasspw/gopass

Crossplatform, git versioning, gpg security (allows you to integrate with smartcards and tokens that you might already have in place for your employees).

Decent UX too, works with clipboard, supports totp.

While I’ve not personally used it, I’m happy with 1Password Teams, bitwarden has a pretty well received project. If you want self hosted teams, though, you will need to purchase a license (it’s one of their money makers, and I think that is completely fair). Most of their product is otherwise open, afaik.

Disagree about the Cloud requirement.

Because: in this scenario you need to drive adoption first,if it is any complicated the humans will not use it. so you need the easiest tool possible.

when everybody is using it, THEN go for the real security move.

(you can try to find a non-cloud one that is super easy, but...)

> Because: in this scenario you need to drive adoption first ...

Be careful: that sort of behaviour can get you fired and potentially land your company in hot water from a legal and/or compliance standpoint, as well as lose them customers.

Compliance is often as much, or more, about having and following documented processes, as it is about having good processes. This can include an approved list of software and services. If you step outside this you are likely breaching your compliance regime.

In the specific case of passwords you may have policies and procedures around secret management that would be violated by storing this information in the cloud, even with a service specifically designed for this. As stupid as it is, the quality (or lack thereof) of your current solution is not the key point here.

If you decide to get all subversive and start using a cloud service then, at the very least, if discovered you're going to get into trouble. At worst, if customers become aware they might leave, and as I've already said, the company might find themselves in legal difficulties.

You may be correct, but in current mindset u would get "What do you mean store all our password with someone else? No." answer. No amount of explaining can fix it.

Bitwarden is fully open source and audited. You can self host the server and client if you want without additional cost (they charge small per user).

Bitwarden derives its keys using SHA256 on the passphrase and a hilariously low iteration count.

The developer refuses to acknowledge it as a security risk, and actually caps the max iterations you can configure, both on the client and the server.

https://github.com/bitwarden/jslib/issues/52

https://github.com/bitwarden/server/issues/589

“It was audited” is meaningless if the audit missed this, or reported this and the dev dismissed it. Either way, it is unsafe.

It looks like people are getting confusing between using SHA256 directly as a key stretching mechanism, and what PBKDF2-SHA256 is.

https://crypto.stackexchange.com/a/61306

Django, uses PBKDF2-SHA256 with 216000 iterations, for example https://github.com/django/django/blob/master/django/contrib/...

Given Bitwarden allows almost 10x that, not sure I see the issue.

Bitwarden’s auth protocol allows the server to tell the client that the iteration count is 5000, regardless of the actual iteration count, and the client will send 5000-iteration passphrase-derived material to the bitwarden server (run by the developer in the usual case).

PBKDF-SHA256 is far too fast, even with the HMAC step.

In the Audit they pointed this out, but didn't see it as a huge issue.

I'd rather they use Argon2id to be as safe as can be.

However, I've not yet seen a fully cross-platform javascript-only implementation of Argon2 at all, which is problematic for an extension.. :-/

UPDATE: There actually exist an Argon2 implementation that's used by another vault, https://www.npmjs.com/package/argon2-browser

Errr...most systems (like Django) send passwords in plaintext to the server! (Over SSL of course).

The point of PBKDF-SHA256 is to protect the secrecy of passwords if the server is compromised and its password hash db is dumped.

Django’s auth system is for a single site. Bitwarden’s master passphrase protects the key that decrypts my password for every single site I use.

This is not some random webapp. It is a password manager.

So use a longer password with BitWarden.

Think about it.

I remember as a freshman in college using Dropbox as version control for a 4-person group project. Even then it was a hurdle. Even with everyone sitting in the same room literally saying "okay I'm working on X nobody touch that file for a couple minutes". Jesus.

I was imagining (wishful thinking, I'm sure) that they were pushing to a Git repo hosted on Dropbox, which actually works remarkably well.

No, he said that they had someone to manually download the last version of each file. If they were really using git hosted over Dropbox they could have simply cloned the repo.

It should be noted that the evidence that stress is a primary cause of gastric ulcers has been fairly weak since the discovery of H. pylori, though there is some evidence stress may be a contributing factor.

FWIW stress is a contributing factor in pretty much all diseases we know. Particularly those where the immune system is active.

I've got ulcer. My last ulcer attack, 5 years had me rolling on the floor. Since then, I avoided stress. Ate at the slightest hint of hunger, even as I grew a pot belly.

Also, I avoided situps, cos the muscle pains felt like ulcer attacks.

There's this specific food that I love. It's eaten RAW but often prepared in a not too hygienic manner. Since I banned myself from eating it, I've slowly become able to stay a few hours without food and can handle more stress.

Seriously not sure if you’re trolling or not.

If not, you’ve made a series of lifestyle changes of questionable value based on terrible (or no) evidence. Avoiding hunger, sit-ups, and raw food has no long-term benefit to your general health; becoming overweight and (maybe) exercising less than before may actually be somewhat deleterious. (Unhygienic food, raw or not, of course may be to avoid... but that’s not related to ulcer formation.)

Ulcers are totally curable via a short course of proton-pump inhibitors +/- antibiotics to eradicate H.Pylori. These are cheap (generic) drugs with a low risk of side effects. Go see a doctor and get yourself fixed up. Chronic gastric inflammation definitely isn’t good.

> proton-pump inhibitors

And when my doctors have no idea what these are? And gave me the, "Avoid these food groups, exercise..."

Get off your high horse. Not everyone in the world has the same level of health care you do.

I apologise if you found my post difficult; that wasn’t my intention.

However, I do struggle to accept/believe that a relevantly qualified doctor, anywhere in the world, wouldn’t know what a PPI is. They aren’t new, they aren’t expensive, and they’re available worldwide. Honestly, every medical student would probably know about them.

Anyway, look for the medications named omeprazole, or lansoprazole, or pantoprazole. They’re PPIs. In many countries they’re even available over the counter. And (was together with a course of the correct antibiotics, where necessary) they cure ulcers.

Thanks I'll check the biggest pharmacy around me for these: omeprazole, or lansoprazole, or pantoprazole.

Currently using Ab Roller for my stomach https://www.youtube.com/watch?v=dzE3Q03KXuY

Just to follow up on this, and in mind of the general risk of taking medical advice from a random dude on the internet...

As said before, please try to see a good qualified doctor about this (ideally a gastroenterologist, assuming this is possible - apologies if this is again an insensitive assumption). The reasons being:

a) If you do have an ulcer, then there are other things (beyond prescribing PPIs) a doctor might help with - such as a test for H.Pylori +/- antibiotics to then eradicate it, if you have it

b) There are other conditions that can present with similar symptoms to ulcers (they can sometimes be very hard to tell apart) which may require investigation to make a correct diagnosis, and which would then need to be treated differently.

c) There are other risks attached to chronic oesophageal/gastric inflammation; if you've had such inflammation for a prolonged period of time, it might be worth being checked out further.

https://www.mayoclinic.org/diseases-conditions/peptic-ulcer/...

> gastroenterologist

Just saw this. I'll do that when I can afford it. For now, it's over the counter for me.

What’s the food?

It's called garri - popular in most (if not all) west african countries. Here's a video of Nigeria's ex-president eating it.

Have you seen this entry in Wikipedia:

https://en.wikipedia.org/wiki/Garri#Health_implications

> Garri is made from cassava which contains hydrocyanic acid .../... can lead to .../... worsening of ulcers.

Thanks for the link.

The ground cassava is left to ferment for some days which removes MOST of the cyanide. There's little to no health control for food in these parts. I've read that over 50% of food stuff exported from West Africa to Europe and US get culled at the border.

See: http://europepmc.org/article/PMC/3074370

See the abstract at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6526674/

When it's prepared with hot water, it forms a solid called Eba. And it's consumed with soup. The heating makes it safe from my own observation.

Maybe you could see stress as taking on "health" debt.

Uh, as much as I enjoy reading this, posting this on Reddit seems particularly unwise for job security purposes. This seems to be an ongoing incident and there's more than enough info shared to de-anonymize.

I’ve come to the assumption that either this happened already months ago, or the much more likely scenario that it’s probably largely fake (if based on a true story)

It mentions December, so either that's a lie, it's recent, or it was December in some previous year.

Updates are written in present tense as actions being undertaken so this seems to be an ongoing trash fire.

But that could also be the day by day as it went in the past, it still fits the explanations I wrote out above.

There's enough info if you are are somewhat related to that incident. For the rest of the world, no.

When Danske Bank aquired Sampo Bank in Estland, Danske Bank spend one million hours on moving and lifting Sampo Bank's IT into Danske Bank's systems.

That didn't work out.

And we all know what happened then.

When the board and ceo want's to aquired a business, IT is just something they believe can follow along easily.

Most developers are straight up lazy and uprofessional to do a job, and mixing in security is just scary as hell.

Danske had some _really bad_ technical ideas: https://hsivonen.fi/sampo-epic-multifail/

haha this is the kind of shit that happens when the engineering team cant get anything from the either the sysadmins or leadership.

Can't run an application? Can't afford a license? Use the network share. No network share? Use dropbox.

And this, kids, is why companies do a technical due diligence during mergers. :0

It doesn't matter. The board and CEO will _never_ listen to the CIO and technical team. If they want to do an acquisition, they'll do it. I've seen this at least a dozen times in my 36 year career.

I keep coming back to from the board and CEO's perspective they have a choice, spend a lot of their time and effort to gain market share by investing in their own company. Or acquiring it through an acquisition.

Seems like an acquisition is a lot less hassle and risk. And can be done quickly. Most 'problems' can be made to go away by throwing money and fixers at them.

This doesn't always work, see HPE vs Autonomy. That case is incredibly interesting for an insight into how idiotic and lazy execs can be. Usually though, it works out for them, probably by luck. Would be, err, "interesting" if there were more Mike Lynch around to provide similar insight (Autonomy's CEO). Then again, the case is interesting as HPE was willing to litigate, and a certain British tech news outlet picked it up.

It is called a "formality"

What a complete nightmare! I bet that most of that data is useless, and the Dropbox account is just used as a dumping ground for everything and there's no tagging, cataloging, or any sort of data structures.

My assumption, given the storage requirements for laptops, is that the actual database is 2-3TB is size and that the rest is archived snapshots of the DB.

Assuming that this database has been building up over the past decade, I'm in awe over how much money they must have spent on Laptop's in the early part of this decade.

i had trouble understanding the laptop part of the story. You can’t sync 450 terabytes to a 4terabytes laptop anyway. So why the big storage ?

It's probably as poster above says: each snapshot (sounds like it's DB + app instance or something, maybe a VM snapshot?) is probably > 1TB in size. If you have a 1.5TB image it would make sense to want 4TB storage so that you could have two versions locally at a time. Also explains how long it takes them to do things. Every time they switch versions they have to swap out TBs of data to pull latest. Because it also sounds like they don't have a way to grab just the diffs. They grab whole images at a time.

Dropbox selective sync that ignores the archived versions maybe?

The best worst thing I’ve ever read. No idea how they pulled that off for so long.

Reminds me of a firm I worked for. They decided to adopt Google Drive for the team of about 20 employees.

Every couple weeks an employee would mistake the synced folder for their local disk and rearrange files or delete files and then the files would disappear for the rest of the company.

Then we’d have to go into the backups and restore the files and send a message out to everyone to hold the files they were working on before saving them to the drive again.

How come? Google Drive has fairly granular access control access control.

That was a cool nightmare scenario story, but holy mother of mercy was it ever tedious to read!

I can tolerate some bad grammar, but somehow the combination of incorrect articles, words auto-spellchecked to something completely different, run-on sentences, and lack of punctuation, made it a real chore to read through without getting lost...

What? :)

I hope the OP writes a book with zero changes to his writing style.

OP, please write that book / collection of stories. Loved reading this, it makes me appreciate the culture and quality of work we have in our company.

What. Personally I really enjoyed the writing.

The author is probably a non native English speaker

It actually reads to me like the author is definitely a native English speaker.

Agreed.

The patterns sound like native English speech.

I suspect dyslexia.

I couldn't make it past the first block of text. I wonder if people who write like that realize how it makes them look? I'd never hire them.

Personally I found this piece pretty fun to read. The guy is ventilating some stress through prose, not applying to work with you.

I enjoyed it too. It reminds me of The Cuckoo's Egg by Cliff Stoll. Both were written in very similar style.

I don't doubt that there's an interesting story in there. I tried to hang in long enough to find out. I can only take so much, though.

Hang in there, man.

i en joyed it but s true that lack of pucntuation and grammatical errors made it harder to read then necessary. /s

Is this a problem with tech? I imagine the users who decided to put sensitive data on dropbox are personally liable? Legal might have work to do? But i fail to see the tech challenge?

It seems that Dropbox was hosting all of the production data and being used as some kind of document store / db system in this story.

Can tech fix it by 1. Disabling the dropbox account? 2. Stopping dropbox urls from resolving in their lan if need be? Or blocking dropbox ips? 3. Buying a md1280 or similar storage quickly n put a freenas or similar on it? 4. Put syncthing or similar on all the user computers n this new nas?

Actually why is dropbox involved if its a git directory???

They probably started out with Dropbox "back in the day" as a way for shared access to things (eg employees in multiple locations), and never put adequate time into moving to something more appropriate later on.

Btw, it sounded like the "version control" mentioned wasn't git.

Maybe they're doing full copies of the application + data into a timestamped directory or similar.

That kind of thing chews disk space. :/

Git annex? Sparkleshare?

Can tech just stop the dropbox bit n just set up sparkleshare for everyone if git is holding up? If git is less than efficient..then syncthing, seafile would all work for their file sharing needs n r trivial to setup?

I think you missed the part where it’s not Dropbox as a file sharing solution that’s the only problem - their entire web app runs via Dropbox API calls from Heroku!

As an employee, you are shielded from all civil liability and some criminal liability. Sure you may be named in a lawsuit but it’s because you represent the llc/Corp and your only responsibility is to forward the documents to your upper management/legal department.

I hate to be That Guy but how on earth is this not discovered during due diligence before “insane asylum” Company was acquired?

That's easy. Lies, misdirection, and the person doing the technical due diligence being a C-Level/Director with zero IT background but plenty of project management experience.

They know just enough to be dangerous and frustratingly useless getting useful information from. So your integration team has to do all the planning with only the foggiest idea of what they are getting into. They then spend the entire first day on the ground just getting the lay of the land. No time to schedule local cable contractors for the really needed cable runs. Let's hope that the existing CAT5 runs work fine on gigabit for the next few weeks till we can get those contractors in to replace all the wiring. Worse case, we'll have to limit the port speeds to 100Mbps at the new switch if retransmits become an issue.

Even more fun when the next project is even less detail but the entity you are acquiring is on another continent instead of a few states over.

Reminds me of HP's 11.1 Billion dollar blunder of buying Autonomy....

https://www.nytimes.com/2012/12/01/business/hps-autonomy-blu...

That article was worth clearing cookies for. Thank you.

Sturgeon's revelation: 90% of everything is crap applies to due diligence.

I just remember two of our engineers had a gig, spent 6 months at $150/hr wading through VHDL and C++ code of some company that was being acquired. They also had to work closely with a patent attorney who was billing probably $500/hr.

Have been there for an acquisition that did not go as planned. Essentially on the small side you are very dependent on the disclosure of the parties involved; short of truly burdensome diligence processes you are sure to miss things if the company being acquired is not up front about things.

Dropbox would really let you get to half a Petabyte ? I know it says as much space as your team needs, but wow. Do Dropbox start to charge per data stored when you take out an enterprise account ?

While it all feels insane from one perspective, it’s not hard to imagine a different narrative where the “founders of ‘insane asylum’ hacked the startup system, were product first, got acquired, and used their new home to pay down tech debt.”

and made everyone else suffer?

Also - they are NOT paying down the tech debt. The programmers and sysadmin staff doing overtime mentioned are.

What are the advantages / disadvantages of going panic mode like this?

Would just increasing the size of the dropbox to keep things muddling along and then trying to figure out a migration strategy come Monday morning be impossible for legal reasons?

Sounds like the big issues are at the compliance level. They’re in a time of the year where lots of companies have much more limited availability of engineering, management, and legal staff. Sure they could easily eat the cost of temporarily upgrading the Dropbox account. But as OP describes, there are huge implications in terms of security.

They didn’t find out until today that a number of senior staff were laid off during the acquisition. Lots of room there for bad feelings. That opens them up to all kinds of vulnerabilities from disgruntled ex employees.

Imagine the repercussions come post-holidays if something bad happens during the next few weeks before everyone is back on board and there’s a security breach over the meanwhile.

This story has many angles, the developer who built the software, the founder who took all the money, then sold the software and fired the developer. Then Dropbox who we all laughed at, is now used for distributed databases.

I don't use Dropbox so I am wondering, is this fundamentally different from storing everything in S3? Is it like a single S3 bucket? Or multiple buckets on the same account?

Dropbox is essentially just a cloud synced folder. Anyone who had the shared login credentials had access to the entire production server and fragments of it could be left on any computers that were synced to it. Anyone could also delete or modify anything and it would propagate the changes to everyone else's copies with very little logging or version control. You can restore a previous version of the file, but it isn't like git or anything... just more like recently saved versions.

Is it shared credentials even for business accounts? Surely they must have some kind of team based shared access option for that?

The post said they had an enterprise account, which I assume could limit access via separate logins, but many of the users were sharing the same single set of credentials (e.g. everyone logging in with the same account), so there was no real access control or knowledge of who all had access using that account.

This looks like how no-code projects will end up in the future.

Reply on news.ycombinator.com