WordPress VIP sites have been hit by outages | Hacker News Reader

WordPress VIP sites have been hit by outages

techcrunch.tumblr.com

76 pointssidyapa26 comments

Open articleView on HN

Here's a plain version without the Oath consent form: https://outline.com/BhZZTZ

This one might be a better article: https://news.ycombinator.com/item?id=20158002

Can't this outage not be fixed? I'm seeing much faster load times!

Updates: https://lobby.vip.wordpress.com/2019/06/11/503-errors-and-av...

I used to work on a VIP site, it was their largest customer and we got 600 million monthly pageviews.

Scaling WordPress to that level is a challenge, but with multiple levels of network and software caching, it worked.

My particular sites wouldn't be down, because of akamai fallback.

i had a site with ~ 10m pageviews monthly that used to run on a $20 DigitalOcean instance.

What kind of things you need to do to scale that up to 600m PV monthly?

Depends on whether the majority of views are on static blog posts, as they can be cached and served from nginx. But if you’ve got a lot of dynamic views - comments, admin, personalised - it’s much harder.

Still impressive you managed to get that level of performance out of a $20 DO box.

10m page views really doesn't look that big if you bring it down to pageviews/second and there's some reasonable distribution.

Assuming a busy day is 1M, and that a busy hour of the busy day has 100,000, but over the hour is evenly distributed, you're looking at under 30 page views per second. Maybe peak is a little more. If this is tuananh's current site, it's Jekyl, so it's all static -- you could probably host that on a 486 (as long as there aren't any big images to clog the nic). If it's something with a little bit of dynamic content, you might need to be careful about how you structure it, but something reasonable should be fine at that level of traffic too, just need to make sure you don't spend a lot of cpu doing useless work (like some frameworks that take 60ms to output hello world)

WordPress has alot of filters that allow you to take over/modify just about anything including internal functions. We overrode get_posts() to use ElasticSearch instead of MySQL and added lots of caching to alot of things with Memcache.

I built a plugin which cached all of the HTML output of every WordPress page and all of the widgets in Memcache.

On top of all the server side caching, there was Akamai in front caching every page at the network level.

Not aussieguy, but cache everything on the network level. If you need anything dynamic, do it SPA style and bypass the WordPress backend.

There's no getting around the 50-200 request/sec barrier of the codebase. The most popular sites use it as a fancy static site generator.

Write your dynamic stuff in React or Vue (something client side). Then point it at a custom backend built to handle the load. For comments and the like, maybe mongo or elastic with Go/Java.

I've only seen a couple sites popular enough to do this.

A danger with WordPress is attackers finding a stray dynamic endpoint (not cached) and taking site down by curling or F5. Unless you have a super beefy server, it's possible to kill a site spamming F5 if you find an action not cached. This is a technical problem for all WordPress sites but only a practical one for sites large enough to be a target.

Some sites the use sitemap functionality to pregenerate all pages as HTML and serve that. It's the only way to be truly secure with WordPress

Rolling Stone are currently on the default Twenty Seventeen Wordpress theme.

https://rollingstone.com/

Quite amusing.

Hackaday is borked too.

https://hackaday.com/

Diversity in nature or technology is important. And it's worrisome that both are disappearing.

"The resulting homogeneity and uniformity can offer substantial advantages in both the quantity and quality of crop harvested, but this same genetic homogeneity can also reflect greater susceptibility or pathogens. Thus it appears the more that agricultural selection disturbs the natural balance in favor of variety uniformity over large areas, the more vulnerable such varieties are to losses from epidemics." https://www.nap.edu/read/2116/chapter/5

- https://www.theregister.co.uk/2018/06/01/aws_outage/

- https://www.theverge.com/2019/6/2/18649635/youtube-snapchat-...

- https://www.theregister.co.uk/2019/06/11/wordpress_vip_go_ou...

And if we take the word of Steve Jobs on this one we are for a hard ride: "It turns out the same thing can happen in technology companies that get monopolies, like IBM or Xerox. If you were a product person at IBM or Xerox, so you make a better copier or computer. So what? When you have monopoly market share, the company's not any more successful.

So the people that can make the company more successful are sales and marketing people, and they end up running the companies. And the product people get driven out of the decision making forums, and the companies forget what it means to make great products. The product sensibility and the product genius that brought them to that monopolistic position gets rotted out by people running these companies that have no conception of a good product versus a bad product.

They have no conception of the craftsmanship that's required to take a good idea and turn it into a good product. And they really have no feeling in their hearts, usually, about wanting to really help the customers." https://www.businessinsider.com/steve-jobs-on-why-innovation...

With our economy and daily lives so dependent in a few companies, any drop in quality will impact all corners of society.

Maximally efficient is minimally robust.

I'm not sure why you think hosting diversity is disappearing when it comes to WordPress. The past 2 years has seen a huge surge in great companies offering premium managed WordPress hosting services.

Liquidweb, cloudways, wpengine, getflywheel, pantheon.

Granted, many use google cloud or similar for actual hosting rather than their own metal... But the services built on top have never been better.

Agreed, VIP is good for enterprises, the paid tiers are okay for personal sites... But as soon as a medium sized business gets into customizations, they have to move off WordPress and onto one of the many dedicated WordPress hosting companies.

Hackaday's case was interesting, they quickly hacked (sorry) a fix together and by mistake ended up piping all comments into the RSS feed.

https://hackaday.com/2019/06/11/the-day-hackadays-theme-was-...

Aren't most of these sites fronted by CDNs that should solve that problem?

WordPress blogs/themes are not static assets.

You can still front non-static assets by a CDN. Just update the CDN when the asset changes, or add a query parameter to requests (e.g, ?v=2) when changing things like your favicon.

They are often rendered once and served until the underlying content changes or some kind of TTL expires.

They kind of are though.

Typical Wordpress sites that are built as a pure CMS are absolutely cacheable (not just JS, CSS, images), but the html can be cached of course. The problem occurs with commenting, and dynamic data features.

We have some sites hosted under WordPress.com's Business Plan; luckily it looks like we are not affected by this.

We host our sites on Pantheon - I've loved them. Check them out - pantheon.io

I know many Pantheon people closely and had seen what they do. It's a good platform, but WordPress.com platform tend to be a lot more mature and reliable, despite this incident.

Reply on news.ycombinator.com