How I built an app with 500,000 users in 5 days on a $100 server
medium.com
medium.com
I really don't approve at all of the GoChat shaming going on. The author may be 100% correct that GoChat made mistakes in writing code that doesn't scale well but that doesn't give him a blank check to beat and berate GoChat. It reads as a very discouraging post to newer/less experienced programmers in my opinion, essentially "Don't even bother making something unless you know it can scale to millions of users" which I think is a terrible message to be sending.
I personally don't feel like he was berating GoChat, if anything I was sad for GoChat to have grown and fallen under it's own weight, but yet still enough respect for GoChat to have had the foresight to build and execute. The author even called it a "genius move"
You don't feel that way because he edited his article and in its original form it was VERY uncharitable to GoChat. Now, after the edits, it's only lying by claiming they're similar in performance profile, when in reality, in terms of realtime performance, GoChat is all GoSnap does PLUS a lot more extra work.
And besides (leaving aside the numerous other 'issues' with the claim, as well discussed here) is "My $200/mo stack performs better than a $4000/mo stack" really that much worse than $100?
Plus the condescending tone when he is just copy pasting code from other project is really annoying to read. Apart from this the content is interesting though.
This is exactly the word I was racking my brain for! I love reading about how to make something more scalable by there was so much filth I had to wade through to get to that in this article...
A lot of condescension for 3 small points.
Now I would use pusher but that's extremely expensive but scalable.
Years ago, you had to suffer using poll(), select(), whilst dealing with threads or forking. It's so easy these days. A raspberry pi should be able to support 1,000 clients with any of the above recommended languages.
The original article is condescending to say the least, that doesn't mean that $4k of ongoing hosting costs shouldn't be able to handle a few million simultaneous users... though, something tells me that Pokemon Go could approach twitter or FB overhead, and related apps like chat could see huge swarms as a side effect.
That kind of problem would kill most first generation apps, as those who've seen twitter grow can attest to. I don't think there are any IRC networks that big at this point.
If you have problems with this, there's something really wrong with either your design or your choice of tools.
How'd you go about building this?
I have used it for multiple projects and it gives a huge head start compared to starting from zero. Signing up, logging in, resetting the password, uploading, etc all seem like easy work but when you pile them all up you can easily spend a week just getting to the point where you are within minutes of cloning the starter repository.
However the failure of GoChat is not relevant to Pokemon Go. While GoChat might have done something very wrong comparing 1mil users to an app with tens of millions of concurrent users is invalid. Pokemon Go would be a NoGo running on a single Node.js machine without any sort of balancing.
That part is almost right, but also a little confused.
He built GoSnaps, which is a geo-enabled gallery with: No chat, few messages, few writers, many readers, medium amount of reads, large messages, not time-dependent. (Not even user state.)
Then he contrasts that to the failure of GoChat, which is a geo-enabled chat with: Many-to-many chat, many messages, many writers, many readers, many read operations, small messages, very time-dependent.
The performance profile between these two things is wildly different and the way he's patting himself on the back over what amounts to half knowledge is pretty disgusting.
Edit: Wow, he did update it and he's only defending his bullshit with more half-knowledge. At this point i'm ready to call him a liar, just over this claim: "My conclusion is that both apps are very similar in terms of scalability complexity."
This is why I don't see the point of moving over to Elixir. Rails has ready-to-go and battle tested "modules" that you can pop in and go.
If I used Elixir now, I would have to wait for the community to build something battle tested or I can home-roll my own Elixir authentication. The Poster above just said, "It's a pain particularly user authentication."
That said. A chat application is really, really well-suited to Elixir.
I think no-one builds their apps this way yet so maybe it needs more explanation.
The grand parent was asking about a template/hackable starter kit with batteries included. That setup of ueberauth is the best I've found sadly.
There's a world of difference in building a photo sharing app with XXX,XXX users vs. building a chat app with XXX,XXX users.
When you do anything that involves chat or that level of concurrency, surprises will bite you in the behind, multiple times, even if you desperately try to use as much existing software as possible.
(as anyone who's taken a look at ejabberd, thought it'll play nicely, and then load tested their code will tell you)
Frankly, PHP vs. Rails vs. Node[1] vs .Net vs Java will be the least of your troubles.
[1] I do fear that the author is going to find a nasty surprise or two for themself regarding Node's performance issues
Frankly, I'm sure Erlang is great, but with 50 full time skilled engineers I'm pretty sure you could run WhatsApp on almost anything.
The hard part is routing messages, and looking at some of the messages, seems to be location based chats... probably could use geohashing for sending messages to channels, the hash level you are at, plus the surrounding 8 hashes.
Breaking up the server instance that a given chat connects to based on location hash would probably work out well enough. I'm not specifically familiar enough with the app... But it seems to me that channel growth and routing are the harder things in this at the scale of millions of users.
I also tend to disagree a bit with the article. For every situation like this were early scalability is important their are a 1000 MVP apps that are prematurely optimized or over engineered. At the end of the day the chance of anyone building an app that will get over 100,000+ in a week (and keep those users coming back) is very very very slim.
This makes 3 months project delivered in 1 month and in the end the client usually benefits that.
> with the argument that one day he could just scale
> it up even by rewriting it.
Which in practice of course never happens. Instead, the old code gets patched up until everything fails completely, and only then a rewrite happens :)
Pokemon Go is a bit of a singularity in growth. It's apparent to everyone on the street that (atleast, for the moment) this thing is huge.
When something is so obviously on fire with growth, over engineering might not be so bad. Look at any of the popular PoGo apps.. they have all dealt with big numbers.. the landscape is just a bit insane right now for that game.
I like making side projects, and i definitely agree with you not to over engineer.. but i feel like this is one case where it may not be the end of the world. With PoGo, if you don't balance the two (over engineering vs mvp) you'll likely end up sad, either way.
Why... would anyone actually do that in anything more than a classroom example for an application like the one described? Amazon S3 and similar services have very decent libraries for pretty much every popular programming language, why would you re-implement that?
>>> MVP and scalability can coexist
I'd replace that with less catchy but probably more correct 'experienced devs can make more scalable mvps with little extra cost, if any'. MVP doesn't mean lets just go silly and make the quickest and dirtiest decision imaginable.
It's a matter of experience to recognize potential problems and the respective potential solutions, and program accordingly. SQL schema is a pretty good example. Often it makes a big difference in scaling and often you can design the initial schema to be much more scalable with some experience and a few moments of planning.
That's why I was really wondering if there are people who find strong reasons to do that.
It sure seemed like the OP skimmed the article, found something out of context, and refuted it here.
He mentioned that he is using Google Cloud instead of storing images in MongoDB and my question was what's the big deal about it? It doesn't seem like something you'd do to make it more scalable, it's something you'd do anyway.
Like just about everything, each has trade-offs.
I don't find it immediately obvious that a database would never be used for such a thing, or that reaching for a third party file storage service is the only solution to consider.
> As said, GoSnaps uses NodeJS as the backend language/platform, which is generally fast and efficient. I use Mongoose as an ORM to make the MongoDB work straightforward as a programmer.
I'm not there's anything substantial that prevents, say, Python+Flask (I see MongoDB and I haven't used it from Django, so avoiding the topic) from handling 600rpm on a machine where Node can. From what I got, all it does is processing messages and geospatial search queries - essentially merely passing those to the DB (which does the actual work) - so how come the runtime of the byte-juggling middleware layer even matters here?
https://medium.com/@tschundeee/express-vs-flask-vs-go-acc087...
(compared to go/java which is 5 times as fast)
A better (and likely more realistic) comparison would be to actually hit the database: node is async by nature so would better handle concurrent requests.
I am not trying to make any particular point here. In my original comment I was just providing a bit more context on whether or not nodejs/python can comfortably handle 600 requests per second, and if so, how much more "scale" is left in this architecture.
In any case, in practice, node.js can process almost as many as requests per second as nginx and PHP in common configurations, after you've implemented clustering - what other servers do by default. It's a dozen or so lines of code in node.js.
I expect the same is true with Flask, except that Flask's default configuration is even worse - Flask's built-in HTTP server is for development purposes only, and is similarly single-threaded. It's a WSGI framework, and deployment needs a proper WSGI server, of which there are a number of choices - gunicorn or uwsgi are common.
Using the default configuration is not a particularly good way of benchmarking the limits of a framework, because many of them are set up to be easy to work with by default, not to be fast. If node.js clustered by default, you couldn't use global variables for state and might have to deal with race conditions. Flask doesn't ship with a web server just because there's so many good choices available depending on what you need.
I cannot even be convinced there is any job nodejs really works on except pass around the requests. Let me put this way, no matter what language or framework he use, spend 90% of his money on MongoDB is the win for this app.
Maybe a slow language doesn't matter too much when the majority of the work is conveying data back and forth from the database to the users
But there is one take-away at least... design your application around your data and how your users will interact with it and performance will generally fall out of that. And it doesn't take much to start that way rather than leaving it as an after-thought.
People might break out the (oft-misquoted) "premature-optimization" horse for a little beating, but performance does matter. At least the bounds matter for most applications. You might not need to eek out every cache line but you can set targets up-front to say, "We cannot tolerate more than Xms req-to-res time" and bake that into your design.
As for pre-mature optimization, for something like this, they should have had a plan to grow, and enough base to be able to handle some early growth. I think falling over once you hit a million or more simultaneous users can happen in a lot of ways. Especially if growth happens faster than you can provision servers/funds.
The author tried to sell us on the idea that Python is slow because Django is slow and you shouldn't use Ruby because Rails is slow. You should just use Node because its fast.
Well I don't know about you but I've seen slow Node applications too.
It all comes down to data. If you really want to maintain your performance goals you have to include them in your design. You have to design for your data. Show me your data and I can write the program. Design it well and you can scale up when the time comes with minimal effort.
Besides there are other factors to consider such as familiarity with the tools, correctness of the implementation, etc. Javascript is great and all but it is incredibly easy to make errors that will go unnoticed until it hits production. The law of large numbers won't protect you if you're used to the blanket of obscurity. And so I find you need many more layers of tooling and choose your subset of the language carefully to maximize its use... something that you don't have to do as much with OCaml or Haskell for instance.
Hence, "just use Node because its fast," is misguided at best. (And I'm not even a Node hater.. I maintain a number of Node applications presently).
[0] http://magic.io/blog/uvloop-blazing-fast-python-networking/
How many times did a project fail, because it the non-aspects (e.g. scalability) were undernegineered? How many times did it fail because it couldn't ship on time/budget due to excessive engineering? We do not normally read such stories, because they are totally unexciting, taken separately. And one failed case of GoChat does not a worthy stat make.
Ultimately, good job to the guy for nailing a sweet spot between his skills and the market of the application created by those skills. Just do not assume that's everybody's sweet spot.
> On the database side, I separate the snaps into a few different collections: all snaps, most liked snaps, newest snaps, newest valid snaps and so forth.
Pardon my ignorance, but don't most databases have some method of handling these issues?
(defining multiple indexes for use, having support for geospatial data, having support for like, subsections of the existing dataset, etc?)
I thought that the main goal was to offload the developer's code's logic onto the performant database, as opposed to offloading the database's logic and caching onto the developer's code? is the former not practical?
Channel position/calculation can happen client side, and subscribe/unsubscribe can happen server-side. Though that may leave room for unscrupulous behavior, it could be locked down a bit more by moving sub/unsub server-side.
The issue will be growth/routing/rerouting of channel data... even then, you can get pretty far with RabbitMQ backed socket.io ... you might need to custom create something before hitting 10M simultaneous users, which at current growth rate would be an issue anyway.
But if you can live with that, then yes, you're unlikely to have actual scaling problems - at least not for projects like the OP.
No need to manage separate collections.
https://news.ycombinator.com/item?id=12122681
https://news.ycombinator.com/item?id=12132605
https://news.ycombinator.com/item?id=12132610
Despite getting a few votes, neither of those submissions got any real attention first time round - no doubt pure chance that this one has got enough attention to hit the front page.
What's been mentioned in other comments but not explained in great detail is the database design, so I want to expand that:
The right way (TM) to do databases is to design a solid schema to keep data integrity and then apply indices and caches depending on your application needs. To be honest his application seems super simple to cache top-down, so a few lines inside the nginx config (which seems to scare him for some reason) would probably do. But if you use a real database (also TM) you can go bottom up, too:
1. solid schema with constraints
2. indices depending on your application
3. stored procedures, database views
4. some non-relational cache like MongoDB to cache denormalized data
5. maybe something in memory
6. (application)
7. nginx caching
He started with 4. What he did is not a solid database design to brag about, instead he hardcoded a cache inside his application. If he wants to scale his application vertically or horizontally he will have big problems, because he misses a point at the beginning which contains the truth on which everything else is build upon. If he starts scaling up and then wants to change his schema he is basically in hell.
What he did is nothing bad. It is exactly "the MVP way". MVP is not about slow or buggy software but a really small feature set and applying YAGNI. MVP is nothing bad, he seems to have great sucess with it! What I am criticising is not how he build his software but what he wrote about it, comparing it to a much harder case and thinking it has something to do with good design.
>GoSnaps uses NodeJS as the backend language/platform
Is NodeJS really that much faster than Python in practise-- even with a fast framework (Falcon, pycnic, hug.rest) and Pypy? I know a lot of work has been put into making V8 fast but I didn't realise it was notably faster than Python.
Let S3 be your database.
You don’t need the full precision of a geospatial query or database if you’re building a simple app that organizes content by location. Depending on your density you segment few 100 meters or few 1000 meters.
How did you market the app ?
In the UK I looked for the app, saw a load of fakers and avoided them but I could see others [children particularly] installing a lot of those apps just trying to find the right thing (instead I installed Ingress to get an idea of how it might all work).
Seems like this sort of inability to find trademarked apps in the app-store makes it look far lower quality over all.
What would have been ideal as a user would be a "get Pokemon Go when available" that would have allowed Niantic to roll it out to my Android device and manage downloads a bit more progressively.
https://www.google.com/search?q=dont+use+mongodb
https://www.reddit.com/r/programming/comments/3dvzsl/why_you...
Do people at big startups use MongoDB in production?
Personally, if I were the author, I'd switch over to PostgreSQL the second I start worrying about more complex queries, though.
Instead he paid $100 for 4 hyperthreads, 15GB of RAM, a few GB of storage and fast but horrendously expensive bandwidth (assuming he used the n1-standard-4, which matches his description).
If he'd set it up to scale the number of servers with load or something it'd make sense but this doesn't make any at all.
Step 1. Find some opensource app code
Step 2. Call it Pokemon Go 2!
Step 3. Upload it to Appstore & link it to dropbox
Step 4. Spend $100 on African "talent" to give fake 5 star reviews & positive comments in app store.
Step 5. Hit F5 repeatedly at Appstore to watch the download counter increase to 5 million in 24 hours.
Step 6. Profit ?!?!
Step 7. Post story in /r/nosleep because too much guilt fooling 5 Million people.
Would love to hear more about what you're doing.
Is it just me or what he telling is rudimentary?
"Don't store images in the database. No, not even then."
4000USD is a huge amount and even 100USD monthly is a lot to spend out of pocket without a plan to recover that money. Do they have any plan of making money out of these sites or are they purely CV/portfolio pieces?
But no need for bashing someone else. These things are a fad so this GoSnaps thing will probably go the same way as GoChat anyway.
It's a bit strange that the author mentions Scala as lean/fast with lots of libraries (along with JS and Go) but Java is too bulky. I'd say modern Java 8 can be used in a pretty lean manner. There's also nice and small web frameworks (Spark etc.).
1. MVP vs. scalability: While building scalable product/s right from the MVP stage is generally a good idea, it may not be particularly beneficial or applicable to most scenarios. I mean
a) how many typical startups happen to scale to 500k or 1M users within days from launch?
b) most founders would be needing an MVP mainly for market validation, as a proof-of-concept and for the purpose of attracting seed/startup funding
c) many founders - especially non-coders - may not have the luxury/resources to have scalability built in to the MVP
2. The original story goes to reconfirm my belief, based on multiple past experiences going back many years, that database continues to remain a (huge) bottleneck for web apps with high traffic volumes and max possible database optimization (right from config tune-up to table structure design/normalization to query optimization) can pay huge dividends in most cases.
Here's what 30 seconds Googling "highly-scalable chat architecture" gave me:
http://highscalability.com/blog/2014/10/13/how-league-of-leg...
https://engineering.riotgames.com/news/chat-service-architec...
http://doc.akka.io/docs/akka/1.3.1/scala/tutorial-chat-serve...
Note: Like to have seen numbers for field-test of the above in the article. Yet, it would've gotten someone thinking.
http://lsirpeople.epfl.ch/nbonvin/dl.php?f=chat-report.pdf
http://doc.utwente.nl/68325/1/imp-pasm.pdf
Previous times doing this for web services led me to highscalability.com with many architectures to imitate with fairly mature software components available. At this point, the common ones should practically have templates for "enter metrics expected here" then click to deploy.
Of course, getting 500,000 users is nowhere near the same thing as keeping 500,000 users.
My server used to be at <0.8% cpu usage. Now that I installed mongodb with almost nobody using my app (< 2 person per hour), my cpu is always at ~1.6% (It doubled because of mongodb!). At first, I feared that my cpu use would be enormous as soon as I would get new users. Now I guess my cpu% increase is due to some overhead that will not grow too much with db size/use (if the author was able to make an app of this scale with mongodb). I'll also try the lean() mongoose thing.
It's poor form to self-aggrandize and say "move fast, make MVPs, etc", and then write a post pointing out over and over how people messed up, when they were trying to move fast and make an MVP.
Pokemon is a rocket ship right now, and any new app has this enormous exposure advantage.
It is also important that it scale well or else you'll squander your advantage.
For what it is worth, I was very impressed by the technical stuff. (It is making me laugh to read about how disappointed others are. I feel like I missed something.)
This is nothing new or insightful, but the author sure found a way to make himself feel good about it.
Depends on your application of course, but if you're self-indexing anyway, may as well lean on your environment (given a proper backup/exit strategy).
Could anyone elaborate on the point the author was trying to make here? is it that erlang doesn't have many pre-existing libraries (for building an MVP) or is not fast enough (or something else)?
You might get a higher acceptance rate.