The Costs of Bookmarking
blog.pinboard.in
blog.pinboard.in
I pay around $3k per month. I own my own servers and lease a full rack and I serve roughly 1 billion page impressions per month. My bandwidth consumption is measured in Mbps rather than amount of data transferred because I get billed using 95th percentile billing. I average around 130 megabits per second of transfer - constantly, peaking at 150mbps I'm transferring roughly 40 terabytes of data per month. 95th percentile billing and owning your own servers is they key here.
To give you an idea, for one month of your hosting bill you can buy 1, possibly 2 servers from Dell and put them in a half rack that will cost you around $800/month including power, secure access, bandwidth, etc. Those servers will last around 5 years with a possible drive replacement or two during that time for a few bucks.
But I think you have another problem that's making things worse. With 15,000 active users you should be able to support them on one or two small Linode servers using round robin DNS. That's a relatively small user base and the number of requests per second can't be anything more than 10 per second? So I'm guessing something about your basic app architecture is off. It could be that you're not using Nginx to reverse proxy to Apache and you think you need more apache children/processes, and therefore more memory, than you actually do. You could have a DB that doesn't have indexes in the right places and so you're IO bound.
I would suggest first looking at your app and seeing where the bottlenecks are in performance. Fix that first, then look at hosting.
Questions:
-How many servers do you currently have and give us a rough idea of config.
-How many app requests per second do you get at peak?
-What's your peak bandwidth throughput in Mbps?
-On your servers, is lack of CPU or lack of disk throughput the bottleneck?
-Have you had problems running out of memory that caused you to buy more servers? Which app ran out of memory?
-Give us an idea of your server config. e.g. nginx -> Apache -> MySQL & Redis. Do the servers talk to each other and if so what do they do?
The most common failure mode in scaling for startups is to have no scaling problem at all.
That said, from what Maciej has written in the past, he sounds competent in these things (i.e. has his databases set up for remote replication and failover), so it would seem that the culprit is more likely over-engineering than under.
No nginx magic in that configuration, either. Just straight Apache, with a few tweaks.
Apache has two stable modes of operation, threaded or one process per child. Both of them require one process or thread per connection busy being served, which requires memory. Apache consumes around 20M per thread or process if PHP or mod_perl are loaded. Roughly 5M if neither are loaded.
If you have keepalive enabled and you're doing 800 req/s and keepalive timeout is say 30 seconds (the default is 300) then you're going to need roughly 800 * 30 seconds apache children to keep up. So that's 24000 apache children or 120 gigs of memory in your server.
A common hack is to disable keepalive which causes clients to disconnect quicker, but they still tend to stay connected for 1 to 2 seconds per request - especially folks geographically far away thanks to latency. So at 800 r/s you're going to need 1600 apache children to keep up, or 8 gigabytes of memory in your server.
Now you could have used an experimental MPM and gotten these results, but I haven't heard of that being used much. So I'm really curious how you managed to get 800 r/s from Apache on a server with 768 Megs of memory.
For the uninitiated: Most people put Nginx or Lighttpd in front of apache which can serve 10,000+ concurrent connections with a single thread. That then talks to apache and connects/disconnects very fast which keeps the threads or processes free and often 10 processes is enough for thousands of concurrent connections with keepalive enabled on the front end.
This is obsolete information: if you are using remotely recent versions of Apache 2.2 (as in, anything from the last year and a half at least: older might even still be fine), you should really be using mpm_event (unless your architecture is reliant on some horribly broken Apache module).
(Also, after having spent years running nginx for this specific purpose, I have to say that it is actually a horrible choice: it isn't smart enough to juggle HTTP/1.1 connections to backend servers, so it burns through ephemeral ports and is fundamentally incapable of warming up its TCP windows... to make that configuration scale in a real "tens of millions of users environment" you end up having to get pretty nit-picky with your kernel-level TCP configuration.)
(Really, though, you should just get a real CDN and drop nginx: a CDN will also reduce the latency of your application from far away locations by holding open connections half-way around the world to your backend, allowing you to drop a whole round-trip from a request that often only requires two round-trips total. I now use CDN->ELB->Apache, and I couldn't be happier with the result: the things nginx was attempting to do are much better handled by these other services.)
Recent benchmarks show apache is a dog with high concurrency, even with the event mpm.
http://nbonvin.wordpress.com/2011/03/14/apache-vs-nginx-vs-v...
Not sure about your nginx criticism. It's the darling of many high traffic production sites. Here's my weekly netstat off a low-end dell front end box on a gigabit link:
(The bottom right number is the interesting one)
My nginx config is pretty stock - standard reverse proxy to apache.
CDN's are gods gift to latency and I use several, but now and then you need an actual web server to do actual work.
Also, nginx being a "darling of many high traffic production sites" does not mean it actually works well for this purpose: if you do a Google search for "nginx" and "ephemeral" you get lots of evidence to the contrary, and you can also prove my statements from first principals of TCP if you really don't believe me: this need not be based on silly anecdotes, you would simply expect nginx to have issues with ephemeral ports due to the way it is designed and implemented (a reverse proxy making new outgoing connections for each incoming one), and if it didn't you would be surprised and probably want to publish a paper on it. ;P
""Compared to putting tornado processes behind nginx, this approach is simpler (since fdserver is much simpler than configuring nginx/haproxy) and avoids issues with ephemeral port limits that can be a problem in high-traffic proxied services.""" -- http://tornadogists.org/1073945/
"""This makes load testing complicated since the nginx machine quickly runs out of ephemeral ports.""" -- http://mailman.nginx.org/pipermail/nginx/2008-February/00352...
My service has tens of millions of users distributed worldwide, making many billions of requests per month to my hostnames. My setup is mostly coded for mod_python (generally considered to be an "older module", especially considering it is no longer even maintained by the upstream developers). I make complex usage of requests making recurrent subrequests through different languages. A good amount of my traffic is SSL.
Of course, most requests are cached at the CDN, so they don't have to go through to my backends, but I still handle way more than a billion requests all the way through to my dynamic webapp every month. These are all handled, eventually, by two boxes running Apache, and I only need two boxes because I want to handle one of them randomly failing (I can easily handle the load on one box: each box can handle, and actually has under previous concepts for my architecture, 3200 concurrent clients).
As for mpm_event in this environment? It works, is stable, is why I could handle 3200 concurrent clients, and you should not be avoiding it because you feel it is "experimental" (yes, even with mod_python). I did run across one or two Linux kernel builds that had regressions that affected Apache+mpm_event (horrible concurrent performance), but you are better off noticing that and steering away from them than avoiding mpm_event.
That said, I want to make it clear that I am not arguing against reverse proxies: I am only making the point that your CDN /is/ a reverse proxy, so there's little point in additionally adding nginx to the setup unless you can't handle enough concurrent connections from the master CDN nodes around the world, in which case what you really want is "just" a load balancer, and you really still want one that is smart enough to use HTTP/1.1 to connect to its backends, and that simply isn't nginx. (Humorously, DNS round-robin, if you think of it as a load balancer, actually works great for this HTTP/1.1 problem, but there are other reasons to avoid it, of course. ;P)
(Now, this said, I heard a few days ago that the just released nginx 1.1 branch now supports persistent backend connections, but I haven't been able to find it in the release notes.)
(Also, as your comment about "now and then you need an actual web server to do actual work" implies to me, but this might totally be incorrect, that you didn't yet notice that a CDN actually provides insanely high latency benefits even if all of your content is dynamic and all of it has to go through to the backend. If you did not know this, you should read my commentary here: http://news.ycombinator.com/item?id=2823268 .)
First, he misconfigured Apache for this environment: by having StartServers lower than the expected minimal process concurrency, they are goading Apache into constantly spawning and shutting down backends (I had similarly spiky performance until I realized this a while back).
However, the setting that simply damns this benchmark is that he has MaxClients set to 150... nginx's equivalent setting (worker_processes) is set to 1024. In essence, he needlessly hobbled Apache, and if Apache does well at all in comparison to nginx, it is probably because the benchmark is flawed. Seriously: this setup is so bad that Apache was returning 503 errors (this is mentioned in the conclusion area) because its configuration told it to stop responding to these incoming requests (and yet, he didn't bother considering that worth examining: he just tossed the detail out there as if it was Apache's fault).
And, given that Apache was less than half as bad (and depending on whether he counted 503 responses in his numbers, possibly "almost as good"), we then do go ahead and question the benchmark itself, just to find that the guy is using Apache bench... Apache bench doesn't actually claim to be very good at highly concurrent testing... specifically, it isn't actually good at swamping remote servers as it isn't itself very efficient. As one random backed-up example, from the BUGS section of the ab man page:
"""The rather heavy use of strstr(3) shows up top in profile, which might indicate a performance problem; i.e., you would measure the ab performance rather than the server's."""
Really, this website's results are based on a kind of "toy" benchmark, and should not really be trusted: the guy (as mentioned in his comments on his post), was trying to go for "the default setup" on these systems, but the default setup is not tuned for performance (this is especially the case with Apache, where distributions expect people to install it on almost everything, including your calculator). I mean, even the nginx settings should be tweaked: in a high-concurrency environment, 64/10096 (where I will admit I haven't spent much time tuning the ratio) would be a much better choice than 1/1024 (the Ubuntu default).
(Reading more of the comments on the benchmark, you can see that other people commented on some of these problems and more, and even wrote entire blog posts responses ;P.)
> ...sustained 100+ requests per second for the last couple of hours; peaks of over 40Mbits of traffic; thousands of simultaneous connections.
So, 1/8th of what you were calculating. Sorry about that.
I use mpm-worker and have keepalives turned off altogether. I also use suexec and fcgid (not fastcgi, unfortunately). I was using mem_cache at the time, but that's off now because it breaks the newest version of Wordpress.
Also, I should have mentioned that this was a static site, so PHP wasn't a factor.
If anybody's still interested in the server config, I'll be happy to share it.
Not all web apps use resources in the same way; you're quite likely making an apples and oranges comparison here (though you don't say what your own web app actually does). It makes sense that Pinboard hits the disk a lot because there's not a ton of shared data -- each user tends to have his own trove of data. Yes, certain URLs will be bookmarked a lot but each bookmark can have different access restrictions, summary data, tags and so on. And no one bookmark is going to make up a big percentage of hits -- you're talking about a ton of bookmarks, each accessed rarely. It sounds a lot more like webmail than, say, a blog in terms of access patterns.
You have questions, and it's nice that you (eventually) asked them and admitted all that you DON'T know about Pinboard. But you should probably ask them before you dole out unsolicited advice, not after.
Instapaper stayed up because the server taken was only a slave. Pinboard had some of their main hardware taken, and while things did slow down, they too stayed up. Half, maybe more of those hosting fees could be servers that do nothing at all but sit around and wait for a RAID.
Would you still be able to build out a hypothetical system as you mention and be able to handle one data center effectively disappearing? Or does that mean taking your information and essentially doubling it, to have a second backup source?
Incidentally, Maciej is banned from HN,[1] due to not being being nice enough to someone.
1: https://twitter.com/#!/pinboard/status/111332316458135553
Maciej seems pretty annoyed by what happened, and I don't blame him. But there you go. Shit happens!
He's basically one of the friendliest people I've met on the Internet, though, so if you want to help him out you can just ping him on Twitter.
He's got a history of similar behavior. He really is a bit of a troll. He certainly doesn't follow HN guideline of "Don't say things you wouldn't say in a face to face conversation.".
My guess is PG only unbanned him so he wouldn't face the criticism of seeming to censor one of his critics.
I thought his comment about Sebastian's post was impolitic, but I chose that word carefully.
I don't care to psychoanalyze 'pg or how he chooses to run the site. He's a busy guy and is more than entitled to make moderation decisions that I disagree with.
I don't understand why you felt the need to write your comment at all. What good did it serve?
He called Sebastian prolix for writing a long post.
He called Sebastian a douchebag for writing that long post.
If we're going to hellban people for writing individual uncivil posts, that's a problem. But I don't think that's what's going to happen.
You should stop calling people "Internet Tough Guys". You have literally no idea who you're talking about. Comments like yours are Part Of The Problem. Maybe you should re-evaluate the notion of starting whole threads on how much you dislike one particular person.
I like reading idlewords and think he's a good/interesting writer and am glad he wasn't permanently banned. But your reply here defending his words seems a little emotional and knee-jerk don't you think?
Regardless, I don't have any vendetta against him or care strongly about him being on the site or not. I realize he's a smart guy who just has a bit of an Uncov streak in him. No big deal.
By the time the doors closed, we had about 16 RU's filled, IIRC. The beefiest DB box had 64gb of RAM in it, which as it turns out is a $1200 one-time upgrade for colo vs $1500/mo extra on your dedicated (quick price checks on newegg and softlayer, I'm sure both numbers can come down).
Yes, making the leap into colo can seem like a big up front cost, but over time these costs do work in your favor. Yes, there is a support cost involved (either with remote hands or you getting up at 3am and replacing that disk yourself). But If your workload and dataset are such that you don't necessarily require a new box spun up in less than 24 hours, the savings could be quite great for you.
P.S. I really hate to point this out, but $30 per month does not inspire confidence. How do you handle backups, and switching over to another machine if the main one should fail (as well as other such items, which normally incur an extra cost in this kind of setup)?
Right now, everything is served from a single server, which is why we get hit by some datacenter maintenance from time to time. Backups are made daily as well as multiple times a week, so that shouldn't be a concern... Let me know if you have any other questions!
I see that your Twitter is rather current, but it wouldn't hurt to do a monthly blog update, even something minor, because I assume that there are other people, than myself, who look at these details when it comes to deciding whether to commit to a service. Said commitment is really less about money, and more about the perceived reliability, stability, and long-term viability of the service.
I suggest that, in the export function, you include everything about each bookmark, such as tags, dates, and whatnot, because it will likely make the customer feel better to know that they can get a complete snapshot of their efforts, whether they are leaving the service, or whether they are attempting to occasionally perform some external operation on their data. Giving customers complete ownership of their information is a good differentiator, because most websites don't provide such a thing.
My last question: How do you handle (D)DOS attacks on public bookmarks? I can see this as being a potential problem.
Thank you for providing an alternative to Pinboard!
Export already includes all the data except the actual page text, so by downloading that file once in a while you can reconstruct your bookmarks almost perfectly (or just import it into your browser).
There haven't been any DoS attacks yet, but varnish is a champ, so that would be pretty easy to mitigate, depending on the scale...
So, it seems that the users paying for the archival service($25/yr) are the one's keeping the lights on at your bookmarking service. 1000 such users would bring in $25K. Basically you need 7% of your users(15000) to break even today.
Each user is worth $2000/15000 ~= 13 cents
My Question:
1. Is it easy to get recurring paying users?
2. In the long run(for any paying web service) what % of the user base do you think will be such users ? [Is 20% the max ? or 30 % ?]
EDIT: Fixed my cost per user
But if they're not paying for archival, are they really the folks who are causing the high hosting costs? As other have speculated, it would seem that the archival service might be the major reason that hosting is so expensive.
That said, it still looks quite expensive.
[1] Since they offer full archiving of bookmarked websites to their premium users.
Presumably he has gone this route in order to keep a large database mostly in-memory, but I'd be interested in hearing more about the application constraints that led to this architecture. Some applications certainly need beefy servers, but they have to provide a lot of bang for your buck to compete with the 10-50 VPSs you could rent for the same cost.
Can you break this down into how many actual servers you have? Is providing a high level of redundancy the source of the high costs?
Why do the S3 numbers fluctuate so much -- that implies a lot of transient data. At $100\month you are storing ~50MB per active user?
Should be spending nearer $200/mo total for hosting a service like this with that number of users IMHO.
Try hosting a cluster of Magento 'daily deal' sites on Amazon infrastructure, with shared storage and isolated RDS instances for each region. You end up with 15-20k $ per month for god damned bloated magento installs.
What I said was that for the functionality pinboard provides, and the number of users using it, the hosting costs are extremely high.
If the hosting costs are high because of inefficient software, or bad architecture decisions, then those should be changed.
If anyone here believes this, make your best estimate as to how many man-months you need for a re-architecture and how many hundreds of dollars you'll save in hosting costs, then do the division to get your effective hourly rate. I'll pay you that plus 50% for contract programming work.
For $21.6k/yr I'd say it's worth a week or two re-architecting.
Yes, it's a different game if you're profitable and $21.6k is negligible, but if you're a startup you should be spending time to optimize things.
The other point is one of scaling. If you're paying $2k/mo to support 15k users, when you scale to 15m users, you could be paying $2m/mo unless you fix things early on.
It looks an order of magnitude too expensive to us, there's probably some simple thing wrong in the architecture.
Maybe we're all missing a key complexity of the service. The Archival service might be it.
As for high hosting bills, build your own equivalent version and then share your own hosting bill data.
Still, easy to criticize without knowing the full facts...
What was your rationale behind these ISPs vs say pure cloud (EC2/rackspace/stormondemand etc)? Or what made you pick each option?
tl;dr - I/O performance