How Did Dropbox Scale To 175M Users? A Former Engineer Details The Early Days | Hacker News Reader

How Did Dropbox Scale To 175M Users? A Former Engineer Details The Early Days

techcrunch.com

49 pointsharryzhang35 comments

This article is horribly written, seemingly by someone with no actual programming experience or knowledge, or even a basic command of English grammar.

I mean, this passage reads like timecube (up until the very end where it becomes somewhat more cogent):

  Out of this came some learning such as, in terms of app
  specific metrics such as ease and ‘fooplroofness’. Plus, 
  it emerged that “most graphs are useless”. Instead they 
  built dashboards to analyze the performance; they always
  put lefts on values (like failed log-ins etc); and they 
  kept some slack such as: extra queries were memcached, and 
  delayed optimization of SQL queries.

Either that or TechCrunch is closing in on the Turing Test.

From the comments, a link to Rajiv's technical writeup:

http://eranki.tumblr.com/post/27076431887/scaling-lessons-le...

which covers some of the points in the TC article.

  My only suggestion for choosing technology would be to pick 
  lightweight things that are known to work and see a lot of 
  use outside your company, or else be prepared to become the 
  “primary contributor" to the project.

This should be put on a header for anyone considering the sexy new thing to come out, be it Node.js, MongoDB or the bazillion JS frameworks etc... Not saying you should avoid these altogether; just be prepared to iron your own wrinkles if you ever intend to become popular.

I think "replace the hard drive" is a terrible way to say it, since a hard drive (or SSD or SD or whatever) is exactly what you need to be able to use Dropbox.

Yeah it's definitely "replace the thumb drive"

For most people I know, it replaced email attachments more than anything else. Previous standard was sending stuff back and forth as attachments with names like Proposal_v2_john_edits_2.0_updated.doc

Totally right as well.

In the long-term, I think you may look back at the 'replace the hard drive' statement as on par with Bill Gates 'a computer in every home'.

If all of your apps save directly to Dropbox your 'hard-drive' and thumb-drive are no longer important, and in that way they are 'replacing' the hard-drive, even if not in technical terms, the sentiment is correct.

In the long-term, I think you may look back at the 'replace the hard drive' statement as on par with Bill Gates 'a computer in every home'.

That's not possible, since your local storage will always be faster than a networked one.

Always? Hard drives don't transfer at multi-gigabit speeds. Networks do.

For most users, their network speed is way way less than multi-gigabit. May or may not change soon.

I guess I should have just said 'gigabit', which is standard Ethernet speed and is roughly on par with hard drives. The point is that the network is plenty fast. Backbone and edge both. The only problem is a few isps.

Err, no. The SATA 3.0 spec is 6.0 Gb/s. Most NIC's might run 1Gb/s locally, but once you hit the WAN you're probably talking about 10 or 100Mbit.

Network storage has always been slower than local, and until we get 10Gb to the home, will continue to be so.

Hard drives can't saturate SATA. And many millions of people already have gigabit service. That's roughly as fast as most hard drives.

It's totally possible. It doesn't have to be faster, just fast enough.

For everybody's most important app, the browser, local storage is already basically irrelevant. That's trained people to be perfectly fine with little network delays. Things like Google Apps show that data and apps can easily live elsewhere and it's basically fine.

In the long term, I think the only question is whether data+apps will live together with the app provider, like Google Docs, or whether data storage will stay generally separate, as DropBox would surely prefer.

There will still be some of us who keep big local drives because we really want to know where our files are and control their movements. But for everybody else, local storage will just be cache. Which means whether it's spinning rust vs flash vs RAM is an optimization detail. And that decision will mainly be made by hardware product designers, not users.

An interesting thing to think about in this context is how the long-term costs of storage are falling. In the last 20 years, RAM is ~4 orders of magnitude cheaper:

http://www.jcmit.com/mem2013.htm

Flash is an order of magnitude still cheaper for now. But if networks keep getting better and services like Dropbox get more solid, that may not matter. If you assume trends continue, then putting half a terabyte of RAM in a device may be cheap enough that the added complexity of flash just isn't worth it.

"In the long term, I think the only question is whether data+apps will live together with the app provider, like Google Docs, or whether data storage will stay generally separate, as DropBox would surely prefer."

Yeah - this question intrigues me. I wonder how Dropbox intend to expose all the useful "filesystem meta data" to all the relevant OSs? How is Spotlight on my Mac going to work if local storage is not cannonical? How am I going to grep an entire "local" git repo if the files are mostly not local?

And I wonder if Dropbox even care about those things? Are they both indicative enough of a "power-user" and that they know the "mass market" isn't ever going to ask those questions, when presented with a "home-machine-less Dropbox" which they can access mainly on their phone/tablet, and occasionally on a shared/borrowed/rented full sized laptop/desktop machine?

Multi-tiered storage is already a well known problem space. S3/Glacier, Apple's Fusion Drive, auto-tiering across RAM/SSD/FastDisk/SlowDisk(/tape?) on enterprise SANS.

You don't always (perhaps even "often") _need_ the raw speed available from locally attached storage. Sure, I'm not going to do video editing or keep my Database indexes or Photoshop scratch disk on internet-connected-storage. But there's lots of files in my homedir that could easily be relegated to "it might take a minute or two to retrieve this file's contents" grade storage…

And Dropbox _already_ has a ton of data on how people are using Dropbox on iPads and similar devices. My iPad _pretends_ to have my entire Dropbox folder locally, but doesn't really. Sometimes that means I need to wait a minute or two while a document downloads over 3G to my iPad while I'm in a café - if that means I get to spend 18 minutes working on that document instead of the whole 20mins I've got available, that's a much bigger win than me spending zero of those 20 minutes working on it.

That assume you're "always-on" the Internet.

Microsoft tried that assumption with the Xbox One, and we know how that went. :P

One of the points of the article was that storing data completely on the net, instead of synchronising local and remote versions, is not in their vision of the future because "users shouldn't have to worry whether they're online or not", so what you proposed is specifically what they're avoiding.

Turns out, Dropbox is _most_ useful to me on my devices with insufficient local storage - having access to all my (Dropboxed) files on my phone and iPad is _spectacularly_ useful. I have no doubt there's some very smart people at Dropbox (and Google and Microsoft amd JottaCloud and BTLabs and any number of established companies and startups I've never heard of) are working out how to cater to a new category of "cloud storage user" - people who _don't_ store cannonical versions of all their files/data on their own laptop/desktop, people with a smartphone and a tablet - but with no "main machine" as such. People who want to access all their files from any random shared, borrowed, or rented machine whenever they need a full sized screen and keyboard, but don't need that often enough to own such a machine.

I don't know what the big deal is scaling up. I have worked in a couple of "big scale" apps and it's really simple principles. At the end of the day, it's all about caching, powerful machines, caching, normal coding skillz, caching. And oh did I mention caching?

I honestly don't think it's that big a deal to 'scale'. Programs like early wolf3d impress me more, much much more. And of course, things like the recent quine relay leave me demoralized.

Dropbox does about 1 billion file updates pr. day across millions and millions of active users, petabytes of data and millions of devices. I think it's very ignorant to think that their problem is just solved by using powerful machines and caching...

Ok. So can you give me a few pointers on topics to read up on how they might scale up? Everything I have read in this field is all about - more machines, split data into several machines aka sharing for nice map reduce queries etc.

Genuine question not snarky. I have even read yahoo's fast website tricks - it's all really hacks and tricks, nothing mind blowing.

What I mean is given some extra time most people can figure things out by some googling and trial and error. This is unlike wolf 3d. Unless you spend considerable time understaning math and graphics you cannot do it.

DropBox can't just easily Google an answer like you would with a Wordpress blog. Their bottleneck wouldn't be web based, so how they load their javascript or if they use image spriting is basically irrelevant.

Scaling a website with exponential growth is very difficult. Fix one problem, two more pop up. It never ends. Oops just spent 50k on SSD's to find out it was a load balancer issue.

Please Bbe specific. I have never seen any real life load balancing code. We just used elb, it was mostly configuration. Drop box uses was AFAIK.

Scaling DropBox is not only about sharding the data; it's a huge operation that requires management of thousands of machines, complex and fault tolerant algorithms and management of a lot of people (from database administrators, to sys-ops, to very smart developers). It's also not something that you can Google the answers to as very few companies have handled 100+ millions of users, billions of daily updates and petabytes of data.

Comparing this to wolf3d is comparing apples to oranges as scaling DropBox and implementing wolf3d are entirely different problem domains . But I would say that wolf3d is a much easier problem that's solved by just one man (John Carmack), while DropBox requires tons of resources and hundreds of people with different skill-sets.

So you're acknowledging that their problem space is fundamentally "one of the two hard problems in computer science", but claim that it's not "that big a deal"?

I assume you've also got a novel new approach for naming things? (Which, presumably is too long to fit in this margin…)

"There are only two hard problems in Computer Science: cache invalidation and naming things." -- Phil Karlton

From https://www.dropbox.com/help/14/en "Dropbox's performance may start to decline when you store above 300,000 files."

From personal experience replace "start to decline" with "not work at all." I don't have much faith in Dropbox's growth if they can't fix this.

I think you are mistaken. Dropbox's growth will not be materially affected by an edge case like this. I doubt even one in a hundred people would want to store hundreds of thousands of files in their Dropbox.

If their long-term goal is to become the filesystem for all your devices, then I think you're mistaken. However, I'm sure Dropbox will be able to resolve this issue before it becomes more commonplace.

bigiain% ls -R ~/ | wc -l 422365

So much for even storing my homedir in Dropbox... And they want to "replace the hard drive"? They've got a bit of work to do yet...

Here's a link to the actual talk: http://www.ustream.tv/recorded/35654239

Can anyone explain what this means?

they always put lefts on values (like failed log-ins etc)

left = y axis

Reply on news.ycombinator.com