10 Million hits a day with Wordpress using a $15 server
ewanleith.com
ewanleith.com
These are all great tips to help you scale, but unless you've got a very small site WP site that is also doing 10M hits a day, there are many more complications. Once I have a little more spare time, I'll have to blog about some other solutions we've come up with.
But I think that's the most likely cause of that kind of traffic on a small cheap VPS, as anyone running a big site without clustering is just being silly :)
Anyways, I'm all for articles like this that help optimize WP sites and get rid of the stigma that WP doesn't scale without cost. Thanks for writing it.
Hope I see a blog post from you about it sometime soon, I think the details around that kind of stuff are far too hard to discover for people trying to learn about it.
But that's all theoretical. There's no way someone running a site with that kind of traffic should be running on shared resources without an infrastructure. When that $15 server needs a reboot, the site is offline, and that would be a significant amount of lost traffic.
Capacity, speed, redundancy and cost of downtime... You really shouldn't run on this kind of architecture.
Now, with 11% more hits! :)
I knew I'd seen a similar post in the past, but couldn't find it.
I'm sure my configurations aren't perfect, but they're a lot better than the defaults they ship with, and there doesn't seem to be a lot of "Here's some sensible settings" discussion on either project's website.
https://www.varnish-cache.org/trac/wiki/VCLExamples
At the bottom are some example templates that are a bit more tuned for production sites and are in use on some relatively large WordPress sites - last August, they claimed 8m pageviews/day on 3 frontend machines.
While I used to advocate W3TC, his support for Varnish purging has been ignored and I published a patch to fix it. Also, there are a number of tuning tweaks you can do - I don't know if you mounted your shmlog in ram or adjusted threading. The Debian packaged defaults are not very well planned - I don't know if Ubuntu blindly accepted Debian's defaults or repackaged with their own settings.
Nginx also includes its own proxy-cache which can eliminate the need for Varnish if you were still RAM starved. There are things you can do in Varnish VCL that you can't do in Nginx without writing your own module if that is an issue.
Since you're not running Apache with mod_rpaf, did you alter $_SERVER['REMOTE_ADDR'] processing? If not, most plugins and even commenting/trackback when used with Akismet would break.
Your worker-processes are probably not set well for nginx and there are a number of other tweaks that can help nginx quite a bit mostly with worker_connections and the rlimit_nofile (which if I recall, the default with a large site with w3tc's object caching would end up causing a bit of churn).
somaxconn might also have given a bit of a boost as you start having more traffic hit the backend. Not sure what version of PHP you get out of the repo or whether the backlog is patched, but, at some point, you'll need to adjust the backlog there for php-fpm - though, you would be well beyond the point of being able to run it on the ECS instance you tested with.
Good job benchmarking and actually including your config files.
Didn't touch shmlog configuration, or threading - everything I changed is on the post. I tried to keep it simple enough, in the end Varnish alone is probably enough for 99% of people.
You're right though, I should look at those VCL examples you posted, and thanks for taking a look through the files themselves, very helpful :)
* Not actually the original, but definitely the best.
I'm not an expert on WordPress internals, but the scene is definitely ripe for a replacement simple due to the quality of the API. WordPress has been good enough for most people for a long time, but it has many weak points.
If it was baked in from the beginning, no problem, but there is so much plugin momentum in Wordpress now that throwing a caching layer on top of it would confuse the hell out of a lot of people who just want to write words.
Blaag and Hyde generate static html, which is about as scalable as you can get, w.r.t. speed.
And then I take a look at this blog post that lists all the incantations necessary to scale Wordpress to reasonable scale and I wonder why anyone should do this.
If you're setting up your own server, installing nginx, configuring PHP, and doing automated load testing, maybe you should also consider rolling your own software or using a different package that isn't supremely bloated.
Very few people have all three of these skills, and it's fair to say that 2 and 3 are not yet solved problems. Rolling your own CMS for any non-trivial purpose is always something that sounds like a good idea until you try it, and then you start hitting all of the incredible idiosyncrasies and speed bumps that other CMS have already solved, even if they've done it poorly.
I'm not exactly defending Wordpress and its lousy code, but in my experience with publishing CMSs, if it's powerful enough to flex to non-trivial needs (Modern WP, Drupal, Django, CQ, etc), then it's probably going to feel like a bloated/complex mess to a programmer, because of all the nuances in the problem space.
Having worked on both sides of this one, i'd rather solve the 'scale this crappy software' problem than the 'Build a useable UX solution that does everything we need and works on mobile and in IE8' problem.
This is a good rule of thumb, but I have found one shining exception, and it is called ProcessWire. If you've never tried it, I highly recommend a look. It's a CMS that essentially offers you a blank slate and a set of simple, powerful tools that let you mold it into something else quite quickly.
However whenever I've had a website to develop I've never seen the point of using one.
If the website is going to be very simple the chances are it doesn't really need a full on CMS. All it needs is an HTML/CSS layout and some content that can come from either static HTML files, a few form handlers and perhaps some parts that my client can update themselves. Most of the time this can be solved by simply creating a part of the site behind a login with a few text boxes that update a database and are then displayed on the site or the ability to create lists of things.
I can create this sort of functionality myself from scratch in an afternoon or so and it is usually much easier to use for the client because it will have less buttons on the interface and be designed around metaphors that they are actually interested in (for example types of cake or whatever). I gave a client Drupal to use once and the result was that they would just call me up every time they wanted an update done to the site.
If it's something non trivial then I'd rather not have to work around a clunky PHP codebase and worry about the plethora of security updates I would have to do when I could just create something much more flexible in Java/Scala/Python.
People dont want to learn a new system, and they are confortable with wordpress.
There are millions of devs and designers also familiar with wordpress - which can mean they can leave you in an instant and get anyone else to make changes or add feature to the site easily.
If requirements are simple then all you generally need is an admin area with about 3-4 links.
Something like:
Change Homepage Text
Show Customer Inquiries
Add Item to catalog
This is much easier to understand to a newbie than "Add Page" , "Add menu item" etc. If they want complicated changes they will usually end up calling me anyway.
I always keep the coding for my simple sites simple enough that any competent developer should be able to figure it all out in an hour or so anyway.
Many people who are not technically inclined do not have time to do much modification to their website themselves, so will generally not bother if you give them something with a lot of power like a full CMS.
The first one is key though -- as a business decision.
They want something they can log into once every couple of weeks and post some minor update too, then possibly consider a redesign 2 years down the line at which point they are likely to want to throw out most of what they have anyway which doesn't matter since they spent maybe $1000 on the whole thing.
If the site provides any kind of complex functionality then wordpress is really no longer going to make sense as a core to build around. Here 90% of the site is likely to be customer forms, order processes etc that don't fit into any convenient pre-existing model. At this point if you build around an off the shelf CMS 90% of the site will be custom plugin code so there will not be a huge benefit to anyone who would take over the code.
It's all about making the business itself a first class citizen rather than a particular piece of software.
You make a good point that if you're just a simple blogger firing off posts in a defined layout, you may as well just hand code html and send static files. I took the point of his post to be more for people who know they have to offer a CMS, and want some basic scaling settings.
Just remember to explain all the caveats that are being discussed here. We're nothing without out integrity. Your client won't be listening to that point though, he'll only be excited he's going to be a billionaire for $15/mo.
> "blog software for the technically illiterate"
According to what metric? Not having the ability (and the huge amount of time) to roll their own blog software, or learn how to use Git and Jekyll? Try and look at the world from a broader perspective. Not everyone is a programmer, but they still have interesting things to say to the world.
> all the incantations necessary to scale Wordpress to reasonable scale
10 million page views a day is reasonable scale? C'mon. No combination of Digg and Reddit and Daring Fireball or anything else will get you even close to this. There is a selection of caching plugins for WordPress that'll get any site on shared hosting to easily sustain those types of real world traffic bursts. For people who do need a million plus page views a day, they're in "good problem to have" territory, and probably have long since acquired technical assistance, or have switched to a WordPress-specific host like WordPress.com, WP Engine, Page.ly, or ZippyKid which has high volume caching already configured for you.
Is it, really? I see that most of the most popular blogs use Wordpress: http://en.wikipedia.org/wiki/Blog_software
What blog platform would you recommend for the technically literate people?
This is actually very easy to do. I just did it. Maybe 1000 lines of code for everything, with proper caching and templating thrown in. The most difficult parts of a blog are the comments and search. If you use something like Disqus for comments and Google Custom Search for search, things become very easy to manage.
Had he run the Blitz test for 10 mins or more you would see the spike in traffic up, beyond what you think a Micro should sustain, and then it plummet to near 0 for a disturbingly long period of time[1].
If you are unlucky enough to have a Micro on a host that is fairly saturated, the performance you get is untenable.
Micro's are not "smaller than" Smalls -- they are a different type of monetization production from AWS allowing you to pay cheaply for little bursts of underutilized hardware.
There is no way (none... not possible, zero) that a Micro would be able to provide the bandwidth and CPU power to host a realistic site doing 10 million hits a day even if everything was straight from RAM.
Read through the EC2 forums for any length of time and you'll frequently find people coming in with reports of their machines "stopping" or "crashing, and totally inaccessible" for minutes at a time with a 100% ST ratio; every time it is a Micro that has been hammer for a bit either through benchmarking or use before the hypervisor puts it in a full nelson and brings it down to the point that SSH connections to the host cannot even be maintained.
-- I would also point out that not only is the CPU time allocated in bursts for a Micro, but the bandwidth is prioritized behind every other instance type (unless you were using a CDN, this would make hosting a typical wordpress site sluggish at best from a Micro -- and again, doing 10 mil hits a day... not going to happen in reality) -- yes you can offload all your graphical assets to a CDN, but now this article is about a $15/mo server and a $4217/mo CDN bill which is a very different article.
Additionally, if you need to use EBS at all in here, the story gets even worse with Micro's (even using something like RDS which requires network I/O in addition to hosting site content is all going to collapse on itself within the first 5 mins of the site's life with traffic like that).
All that said, the tips in the article are great. I only mean to clarify expectations with the use of Micro's. A whole swarm of them grinding through a work queue in random order is great; using them as the backbone of your web presence will have pain points. (Yes you can put 20-50 of them behind an ELB, but at that point why not run a handful of Mediums or a few Larges).
Anyone with a Blitz.io account, please feel free to setup this exact configuration and run the benchmark for 1hr with the same load to verify the meltdown.
[UPDATE] Ewan, I am not knocking your article, I wouldn't expect most people to be familiar with the ins and outs of every cloud provider. The tips and techniques are great regardless of the actual performance on a Micro (and applies to anyone trying to scale WordPress). Just wanted to clarify for anyone getting excited that they can now run their Fortune 500 website on a single AWS Micro that there are nasty surprises lurking just under the surface.
[1] http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/co...
I only picked the micro because it's cheap, and AWS let me fire one up, build the config, break it, trash it, and start again, all in rapid succession.
One thing though, this configuration doesn't actually need much real CPU or disk resources - it's pretty much all memory, and as far as I know, AWS doesn't overcommit RAM. this means it should be "relatively" stable. The CPU usage is at around 5-10% even at the peak.
Personally, my own blog runs on Linode, because I think EBS is broken, but each to their own :)
Months after and they still haven't said what happened or what they fixed during the Bitcoin theft fiasco.
I wouldn't trust them again with ANY site (former customer).
It's March 31. slush's post was on March 1.
One trick that I used when I was temporarily hosting a small Clojure web app for a customer: I ran the web app using "nice" to reduce its priority. I did not do any measurements, so this is really subjective, but the app seemed to run a lot better as far as consistently getting a little bit of processor time to run.
Not so subjective: I did some Clojure development in a repl on a micro instance (because it was already set up for access to a Hbase cluster) and doing a "nice lein repl" really made development possible, if a little slow.
That said, Smalls and up on EC2 will give you more consistent performance (they aren't built to provide the spike-performance Micros are) BUT are still relatively subject to "noisy neighbors".
The smaller the instance, the bigger the impact felt; it is still possible to have a small grind to a halt because of a noisy neighbor and typically I do not see people hosting web front ends with smalls because of the erratic and poor performance (in general). Possibly a fleet of them behind an ELB, but even then I see Mediums and Larges much more often as the "web server".
The mediums provide an awesome (and typically unsung) balance between monthly cost and performance on your way to a large (which can get quite expensive).
A million visits times 500KB = 476.84GB. Per day, so 14TB per month. At best that's another $1,600 a month.
WordPress itself isn't the scalability problem anymore, it's usually how the blog is being used, and when the db needs to be interacted with.
At the same time, a lot of our customers use WordPress as a CMS, and have static home pages and sub pages, which we serve in a similar fashion.
https://github.com/zippykid/php-varnish the plugin there, will be handy when you actually start making blog posts.
Taking the least denominator - $50, if he spends just 5 hours a month trying to keep his server alive, active, update and making sure it's not down, responding to it if anything happens - that will cost him $200 + $15 per month to maintain his setup.
If you're about running a well maintained popular Wordpress powered blog, why go to the extend of doing it yourself when you can spend a little more, let the people who runs such services handle it. That way, you can keep doing your more productive work.
Unless, of course, he does nothing else but earns via the website, does it full-time and is happy spending his 4-hour a week on running the server.
But as a learning experience for me, I think this has been pretty priceless, and I enjoy it, bizarre as that probably sounds to the people who don't read HN :)
So yes, I agree with your sentiment, but once you have servers up and running with good tools they don't take that much effort.
Now, I use Page.ly and CloudFlare in the front (WPEngine is an equally good host and I'm considering for another Wordpress Setup of mine) and I don't even care about the host, I just care about my blog and how to update it with articles. It's costlier but it comes with it's reward.
Hell, it takes 15 minutes to set up.
On a normal website 10M hits will not be evenly during the day this example hits are only an avarage of 10Kb, and the number of concurrer users are only 250.
Also the test seems to be done on a new wordpress installation. The more content you have, the bigger the database,the longer the comunication with your DB, the bigger the cache, ect...
In a real world scenario you proabbly would need something more than a $15/month virtual server to handle 10M hits in a single day.
Still optimization is always a good thing.
We did something similar for Wordpress [2] and plan to ship it in 12.04.
I'd love to bring you on board so we can compare configs and make it even better. Also rkalla is correct, micros are great for prototyping but you'll need at least smalls for production. The nice thing about micros is that you can set it up, tweak, and then later reboot them into larger instances, so they're nice for playing, but I wouldn't run a live site on them.
[2]: http://www.jorgecastro.org/2012/03/18/redeploying-omg-ubuntu...
Shouldn't database be more like an add-on, not the core? Sure search is something that is hard to do with flat files, but everything else should just use files. It might be a good idea to save all the data also to DB, just in case you want to do some markup changes (which happen like once a year or something). But querying DB every time someone visits you blog? Crazy.
Also, when you don't need a database, backing up your whole site and/or transferring it to another host is a lot easier.
More complex sites than a normal personal blog is of course a different thing.
Shouldn't that be "oneiric" since you're using Ubuntu 11.10? Or does the nginx team compile everything statically so it works no matter which version you choose?
I'm also a bit puzzled with your decision to make PHP run as the "nginx" user. You probably did this to match the username that the nginx debs use (Ubuntu's default package uses www-data), but what's the benefit of matching users there? If you're going to change it anyway, why not make PHP run as "php", for example? Some might even say that running both PHP and nginx under the same user reduces security.
Of course, you can manage this in other, more security conscious ways (move static assets elsewhere, use group permissions, etc.) but this is probably the simplest.
When you load in a sizable theme, warm up the DB with thousands of posts, tens of thousands of comments and users, and have requirements of short TTLs for new stories to be posted, comment administration, etc, and real world traffic... the picture looks a little different.
I would say you should dump the caching plugin, however, and just do everything it's doing in nginx itself. My mix also adds CloudFlare as a caching system:
http://danielmiessler.com/blog/how-to-run-a-wicked-fast-word...
The only way to make WP "fast" is to make it nearly completely static.
Get a half dozen people working admin area and your server will cry.
Keep the original setup and have your test script perform some random searches. New figures?
(Everything is fast/non-server intensive when you're serving static data)
This rush generated 7,363 successful hits in 1.0 min and we transferred 56.39 MB of data in and out of your app. The average hit rate of 117/second translates to about 10,159,286 hits/day.
So you're right in that Varnish is the big improvement, but the CPU of the server seemed to be significantly higher with varnish alone than with varnish + APC.
Of course, one issue with these systems is there's no such thing as vanilla, the documentation goes from "Install" to "Read 100 pages of stuff to get a working configuration", with nothing in between...
If you want to see how many queries Wordpress runs even for a simple post install this plugin I wrote at my last job. It shows a log of queries in the page footer if you are logged in as admin.
You're right though, if you start adding a few less well coded plugins, then you can start hitting the database a lot, for no really good reason.
>
>cd /tmp/
>wget http://nginx.org/keys/nginx_signing.key
Verify a package with a key you got over http? Am I the only one who noticed this?
Question: If you have Varnish running as a cache, should you really have the WP 3W Cache plugin running too? Seems redundant, but I'm not familiar this tech.
WP caching speeds up requests that spin up PHP by preventing them from making a connection to the MySQL Database. This is useful for requests that may peruse multiple blog posts, but load the exact same sidebar content by caching things like post counts, categories, tag clouds, etc.
It's better to use the language runtime for more specific caching inside your application logic and for smarter cache expiration.
Obviously though, there's cheaper options than AWS out there - Linode include 200GB of transfer per month in their $20 option.
I mostly just used AWS because I could start the server, build it, and blow it away, then restart, all without any hassles or full monthly chargers.
As long as you aren't serving lots of images, this will keep you well below $20/month - even with large amounts of traffic.
If you are serving lots of images, or worried about excessive bandwidth: Linode gives you 20GB/200GB (storage/bandwidth) on a 512MB server for $20, and MediaTemple gives you a 512MB server with 20GB/350GB for $30 (a little cheaper if you are going to be in the 300GB range when you consider Linodes .10/GB for overages).