Wordpress.com serves 70k requests/second using Nginx

Same here - interested to see that config :) --

On a side note - you're completely right - whoever use nginx as a reverse proxy should just stop doing so and transition to nginx and drop apache.

It's not that Apache isn't good - it's just that Nginx is just way better.

Edit: actually, it looks like this doesn't work the way I think it does ... looks like nginx has changed enough between 0.7 and 1.0 that my original config doesn't work as correctly as it used to.

Edit 2: Actually I think WP Supercache is the one that changed.

Edit 3: This config looks more up-to-date than mine -- http://rtcamp.com/tutorials/nginx-wordpressmultisite-subdoma...

I particularly liked how they used the try_file directive to cut down on the if statements.

----

Like most people I've cobbled together what I found through Google searching. My case is further complicated because I run a multisite configuration and use the WP Domain Mapping plugin.

It works in a few stages.

WP Supercache is configured to gzip pages to disk. It used to be that you needed to add extra code by hand to support domain mapping, but WP Supercache now cooperates and puts on-disk cached files for each blog in its own directory.

Then I put specific directives in a sites-available/ config file, not the main Nginx file (I serve a static site off the same server).

First,

    server {
        listen <ipaddress> default_server;

The default_server directive means that if another listen directive doesn't pick up an incoming request, it will be handled by this config. Otherwise multisite goes kerflooie.

I like my logs to be divided by site, so:

        access_log  /var/www/log/wordpress/<sitename>.access.log;
        error_log   /var/www/log/wordpress/<sitename>.error.log;

Getting end-to-end UTF8 on a stack is a hassle because you have to do it in the database (multiple times, MySQL has about two hojillion charset configuration dials), in PHP and then in Nginx:

        charset utf-8;

Then the obvious:

        root /var/www/wordpress;

        error_page 500 501 502 503 504 = /50x.html;
        location = /50x.html {
            root /var/www/wordpress;
        }

Now for the meat:

        location / {
            # Add trailing slash to */wp-admin requests.
            rewrite /wp-admin$ $scheme://$host$uri/ permanent;

The rewrite is just a little tweak. If you go to /wp-admin, Wordpress will often redirect to the home page. If you go to /wp-admin/, it does what you expect. So this rewrite just adds a trailing slash.

            index  index.php;

For multisite, the Wordpress coders change the URLs files are served from, for reasons that are simply beyond my mortal comprehension. Anyhow, you need a rewrite rule for /files/ URLs:

            # Redirect /file/ URLs.
            rewrite    ^.*/files/(.*)    /wp-includes/ms-files.php?file=$1 last;

Then what follows is based on the common config file you'll see on a dozen blog and forum posts if you google around.

The next thing to do is try and serve a file straight off disk. Nginx does things in an unintuitive order, so even though this rule is further down, it will often fire first:

           # Rewrite URLs for WP-Super-Cache files
           # if the requested file exists, return it immediately
            # this covers the static files case
            if (-f $request_filename) {
                expires 15d;
                break;
            }

The "break" directive basically says, "Oh you found whatever.css|jpg|js? Just serve that up kthxbai".

Now we proceed to determine whether or not we'll try to serve a cached file off disk:

            set $supercache_file '';
            set $supercache_uri $request_uri;

            # don't interfere with POST events (ie form submissions)
            if ($request_method = POST) {
                    set $supercache_uri '';
            }

            # Using pretty permalinks, so bypass the cache for any query string
            if ($query_string) {
                    set $supercache_uri '';
            }

            # Don't show cached version to logged-in users
            if ($http_cookie ~* "comment_author_|wordpress|wp-postpass_" ) {
                    set $supercache_uri '';
            }

If there's still a supercache URL, we try to serve it straight off disk:

            # if we haven't bypassed the cache, specify our supercache file
            if ($supercache_uri ~ ^(.+)$) {
                    set $supercache_file /wp-content/cache/supercache/$http_host/$1index.html;
            }

            # only rewrite to the supercache file if it actually exists
            if (-f $document_root$supercache_file) {
                    rewrite ^(.*)$ $supercache_file break;
            }

Otherwise, give up and let Wordpress grind out the file:

          # all other requests go to Wordpress
          if (!-e $request_filename) {
              rewrite ^.+?(/wp-.*) $1 last;
              rewrite ^.+?(/.*\.php)$ $1 last;
              rewrite ^ /index.php last;
            }
    }

    location ~ .php$ {
        fastcgi_pass  127.0.0.1:9000;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME /var/www/wordpress$fastcgi_script_name;
        include /etc/nginx/fastcgi_params;
    }

From the Wordpress wiki I picked up a few nifty directives to make log files a little less cluttered; I've thrown those in a common file which I include here:

      include /etc/nginx/clean.conf;
    }

clean.conf looks like this:

    # Global restrictions configuration file.
    # Designed to be included in any server {} block.</p>
    location = /favicon.ico {
      log_not_found off;
      access_log off;
    }

    location = /robots.txt {
      allow all;
      log_not_found off;
      access_log off;
    }

    # Deny all attempts to access hidden files such as .htaccess, .htpasswd, .DS_Store (Mac).
    # Keep logging the requests to parse later (or to pass to firewall utilities such as fail2ban)
    location ~ /\. {
      deny all;
    }

    # Deny access to any files with a .php extension in the uploads directory
    # Works in sub-directory installs and also in multisite network
    # Keep logging the requests to parse later (or to pass to firewall utilities such as fail2ban)
    location ~* /(?:uploads|files)/.*\.php$ {
      deny all;
    }

The final step is to add this magic directive to your master nginx.conf:

    gzip_static on;

This tells Nginx to look for a file with a .gz extension. So when it gets redirected to look at whatever-blog.com/index.html, it will also check for index.html.gz. If it finds that file, it will serve it instead.

The problem was not that lighty gave a 500 error for transient unavailability.

It's that it got stuck in 500. I would have to log in every week or so and restart it when this happened.

It's slightly nicer than stock MySQL and I can more easily get Percona to help me if it goes bang (it has far more stats available than stock MySQL, for instance).

Some of my sites have sufficient write activity that using XtraDB (basically a slightly souped-up InnoDB) is a smarter option than MyISAM. For searching people can just use Google, it does a much better job than MySQL's inbuilt fulltext search.

I don't know if they've fixed it, but AFAIK tables with full text fields used to cause MyISAM joins to go to disk, even if the full text field isn't included in the query. As you can imagine, this sucks. Maybe that's been fixed.

We started using Percona's MySQL builds by default a couple of years ago. There are some nice performance and convenience features not included in the MySQL.com builds. (Un)fortunately we still have quite a few MySQL 4.1 instances happily running and are still using MyISAM across over 360 million tables. Most of that stuff is not running Percona.

Not sure in this environment, but I've been using Percona builds of MySQL for the last couple years and it's a great drop-in. Good optimizations for InnoDB Oracle-owned bits.

> You can also run a script to watch for PHP/APC segfaults and just restart the the cache.

... wat

> I cannot imagine running PHP without an opcode cache, you are losing a speedup of 300% to 500%

That's just not what my profiling showed.

The majority of our transient bugs on WordPress.com are probably related to APC opcode issues. We try to catch them and handle it gracefully, but it isn't always so easy.

Isn't there a known fix for this for like a year or so? (http://www.justincarmony.com/blog/2011/01/24/php-nginx-and-o...)

Varnish caches the rendered html and stores it in disk for os to cache like mongodb with mmap.

Wordpress super cache plugin generates .html files from the pages and saves it in the disk for the os to cache.

I know they replace the database class on wp.com to support replication (I think they even released the code once) but I don't think they do extensive changes otherwise unless you've actually read otherwise.

It's actually not that different, and of course they make as much use of caching as possible.

So does my self-hosted wp blog, for that matter. If you are using WP Super Cache correctly, you can deal with significant amounts of traffic. I've seen peaks of hundreds of simultaneous users on my blog, with a server load < 1. The server is a modest aws small instance.

APC+mysql cache (not just query but keycache too) is a must, but memcached is pointless unless you have a multi-server website (on a single server APC shared memory is way faster than memcached).

All logged out users should be served a completely cached static page that bypasses PHP entirely, it's way faster than even an opcode cache and way less load.

WP-Super-Cache can serve static pages to logged out users and makes this rather easy to setup. It's practically a must for wordpress.

`service nginx reload` will do the trick as well.

I don't think that is right, it says that automattic has 2000 servers but they also run things like gravitar, akismet, and vaultpress. Which makes me think that using that number is completely wrong. (Also these are just the load balancers not the back end, which could be taking up a large amount of that 2000 server count)

From the article it seems more like they have ~100, maybe less, for the load balancers which comes out to 175 requests/second per cpu which is getting a little more reasonable.

They have two thousand servers total; that includes database servers, memcached servers, regular backends, and load balancers.

Peter Westwood gave a talk on WP.com's infrastructure in London in January; I've tried to find slides online but I can't, which is a shame because he went into quite a bit of detail about their nginx/HyperDB/memcached/mogileFS setup.

It says 12 datacenters distributed around the globe.

I think we can safely assume each datacenter holds a dedicated load-balancing server. Let's assume they have only one of these at each datacenter, it gives us 70000 / (4 * 12) = 1460 requests per second per CPU, which is in line with reliably handling over 10,000 request per second of live traffic to WordPress applications from a single server.

and

In April 2008 Automattic converted all WordPress.com load balancers from Pound to NGINX. , which points to the fact that they're only talking about load balancers and they have several of them.

This link implies that one NGINX load-balancing instance cannot handle 70k r/s.

2000 servers? My god. Aren't most blog posts just static pages or generated static pages? A good html cache setup should help a lot.

I was not aware of ip_hash. But as per documentation it uses first 3 octets as hashing key, this may not work for me.I want to use nginx on LAN. It will throw all clients to same backend server.

http://nginx.org/en/docs/http/ngx_http_upstream_module.html#...