My experience with WPEngine
chester.id.au
chester.id.au
Your staging server does not use WPE's caching technology, which your production server will use. WPE will not give you details on how their secret-sauce caching works. This means that things that work on staging will break on production. WPE also refuse to turn off their caching on production, but also refuse to turn ON their caching on your staging server. This means there is no real way to test. Things that work with their staging server, and even work with the standard WP caching plugins, will break in WPE production.
Their solution? Buy another production site to test on.
That website is now moving from WPEngine after having its launch botched by the aforementioned issues, and after not being able to make a basic paywall work with their caching. Once they realised the use case, WPE simply suggested moving elsewhere without making any further effort to accommodate.
I had a similar reaction to the post yesterday by asmartbear as Jaques, but he made the extra effort to actually write the blogpost, so bravo to him. WPE enjoys this sterling reputation that misleads a lot of people into using them, but their comfort zone it seems is limited to vanilla WP installs with little deviation.
... wow.
Your frustration with their caching "secret sauce" is understandable. However, you can get a rough idea of their caching back-end by poking around in the WP Engine config files and plugins bundled with your installation. They also provide a nice little overview here: http://wpengine.com/our-infrastructure/.
If you are staging out anything more complex than a single blog, you will definitely want to have two production instances to test on. For some, this is a raw deal. I personally don't mind.
I am also not a huge fan of their CDN feature, which works by rewriting URLs in your page source to point files and assets to NetDNA servers. However, I am hopeful that they will improve upon it.
That said, I have enjoyed a platform that keeps my Wordpress installations secure with automatic upgrades.
Comparing WPEngine to other hosting providers like Linode or Dreamhost is really unfair. They offer completely different services, and specialize in Wordpress-optimized hosting only.
I also have to say that it was really nice to not be part of Amazon/Heroku's outage this week. WP Engine has a control panel hosted on Heroku, but all their sites live in the Linode network, evading downtime.
Same basic proposition: fully managed Wordpress.
They cost less.
And their service roflstomps WPEngine's.
Just because you have a bad experience it doesn't mean that it sucks. My experience with WPEngine has been fantastic.
But I wasn't asking for anything exotic. A multisite network, not very large, not much traffic. It says, right there on their site, that they support multisite networks.
So I guess I ... well ... expected that they would be able to support a multisite network.
What blows a story like this up (as much as hitting the front-page of HN is 'blowing up') isn't that a service failed to meet the needs of one blogger, but rather that the people running the service respond to criticism (in public) sounding more like petulant children than professionals.
As someone not into blogging, I'm not that familiar with WPEngine, but based on comments here that link to various twitter comments now I suddenly have a negative-leaning opinion of them. Primarily because they couldn't take a negative review without calling the poster a "hater". (FWIW, a lot of my negative opinion is tied up in this stupid word, which IME comes out as a last resort when the person using it knows there's a lot of truth to what the other person is saying but they refuse to acknowledge it for whatever emotional reasons).
WPE has been great so far -- very responsive support -- and their caching handles about 8 million pageviews a month with 600-800 simultaneous visitors and very little downtime.
1. Email FireHost with the event IDs and a request to unblock.
2. FireHost doesn't reply for 24 hours, or if they do it's with "we can't find that event in our logs".
3. I open a Page.ly ticket
4. "El Vaquero Furioso" (yes, that was his name in Zendesk) replies that the issue is fixed and won't happen again.
5. I ask for the root cause and what steps were taken.
6. Crickets, or a snide response from the founder that "we're all on the same team".
I guess it's like how, in this thread, lots of people are saying they've had great experiences with WPEngine and then there's me, suddenly internet-famous as that unhappy WPEngine customer.
Sorry we handled that poorly. At the time Firehost was having some significant issues with their firewall policies. We do constantly walk that line between enforcing security at the edge and annoying customers that may run into WAF blocks. It's a delicate balance.
The problems you had have since been resolved and pagely is running under a new firewall (and new blades) which are much more reliable.
In addressing your frustration my comment of 'we're on the same team' was to demonstrate that we were all working towards a solution for you. As professionals dealing with other professionals we aim to acknowledge and address the problem while steering the conversation to a positive outcome. If our customer views us as an adversary it makes the process more challenging for all involved. By re-framing the conversation as we are all in this together, all working towards a solution, we feel it is more likely the positive solution will be found and implemented sooner. My comment was a gentle reminder to that effect: acknowledging the issue, and we are working with you, and with our partners to remedy it.
Thank you for your past business, and we hope one day to earn it again. - joshua strebel, founder pagely.
It would make sense for them to offer a dev plan with every production plan though, and let you upload your site, and serve it to a limited no of people as a testing ground. Given their 'Power Tools for Power Users' stuff on their website, they really should consider at least offering staging servers for testing.
I guess at present they're just not a good setup for developers trying to modify WP, and are instead trying to capture the market for hardened WP hosting for mostly vanilla installs.
I don't know of any large WP installs that would need the benefit of outsourced WP hosting/experts that use 'vanilla installs'. They may exist, but I can't imagine they're a large portion of the market. Perhaps my scope of experience is too narrow in this respect?
We invented this feature in response to bloggers and site owners who wanted to be able to test a new plugin before adding it to the live/production version of their site. We're also the only managed WP platform to offer such a feature as standard with all accounts.
I can see that in a true dev/staging/production test suite environment, it does't stand up as a true mirror of product for the reasons you stated - but then neither does the staging/local version of Google App Engine or Heroku offer the same exact environment of their production platform.
If you need true like-for-like mirroring of production and staging, then yes, you need to create another WordPress instance - that's the case for most providers. Also its wroth nothing all WP Engine accounts, other than the Personal plan, come with multiple WordPress instances, so rarely that means actually buying anything further.
You may be the only managed Wordpress platform to do so, but App Engine has had this functionality for a long time.
Not saying it was right or wrong, it just wasn't a fun experience thinking I was going crazy. And hopefully if anyone else is trying SETCOOKIE() and it's not working they also realize they will have to use JS.
Don't get me wrong, as a contractor to a company that uses WPEngine, we have had a great experience with it - but e.g. in the case of setcookie() being completely useless for unauthenticated users, some documentation / warnings / big red flags would be nice.
In my case I was trying to build a simple site selector for unauthed users that would forward them to their selected site every time they hit the main domain. Nothing exotic by a long shot.
This isn't something specific to WPEngine, this is how all the good WP caching plugins work.
Part of their caching setup was causing the API request to fail for a select list of bots. The issue was escalated to a Sys Admin who figured out what the problem was. He tweaked my server and fixed the issue for me within a day. I think that is pretty good.
My point... if something is broken and you think it might be WPEngine's caching. Try it on your staging server.
So, while I'm here, go look at my very nice bloggers. I regularly submit their stuff.
Take, for example, this recent extremely intelligent discussion of privacy in the current internet:
http://skepticlawyer.com.au/2012/10/19/manners-cost-nothing/
Or this discussion of the roots of order/chaos good/evil dualities in agricultural society: http://skepticlawyer.com.au/2012/10/16/war-and-peace/
This confronting moral conundrum from the Massacre of Srebrenica: http://clubtroppo.com.au/2012/10/24/srebrenica-should-the-so...
This discussion of the deadening effect of media laws upon Australian free speech: http://catallaxyfiles.com/2012/10/24/the-deadening-of-free-s...
and so on and so forth.
I am enormously proud of all my bloggers.
How are these blogs related to you?
Good for him for sponsoring people, but it's a bit odd to imply he's deserving of any credit for their work simply for hosting a Wordpress instance. If they are paid employees, then that might be more appropriate, though still odd.
But what I said is true. It's a shame that out of all the things that every site on Ozblogistan has ever posted, this is the one that made the HN front page.
There are far better posts than mine in those blogs. Is it wrong for me to be proud of them?
It's not wrong to be proud, but I have to agree that blatant advertising is distasteful no matter which way you slice it.
In case you are wondering, I don't make any money from the traffic. Each of the blogs decide for themselves whether to host advertising.
I sometimes forget that because I have about 3k advertising servers blocked in my hosts file ...
Personally, I downvoted the comment and suggest you delete it.
Jason, if you're not scrambling to deal with this mess, you will be soon. And the main message I have is that you need to back off the marketing side a little and let the technical and customer service side catch up.
Godaddy makes an incredible amount of money, but they're still a terrible company from a technical and customer service perspective. But their marketing is so effective that they can get away with that [1].
I wonder if Jason Cohen's position in the tech community and general marketing prowess has allowed WPEngine to get ahead of themselves? I know when I read Patrick's blog post about WPEngine, I was really excited and it was a service that I wanted to like. And I do like them, but I'm not excited about them at all anymore.
1. Interestingly, Google is the exact opposite, and their customer service makes me want to take a spoon to my own eyes.
Another founder contacted me by email and apologised. It was dignified and mature of them to do that.
And so I consider that matter closed, which is why I am not going to name names here.
The contrast is unflattering and it's a shame.
I really, genuinely, wanted to like WPEngine.
While perhaps the issue ended up mostly being due to a Wordpress core issue, I question why this was an issue for a service that someone is expected to pay $250/month for?
The OP was able to, without a team of engineers/admins get the same content running on a VPS with no issue. I then question exactly what one gets when they pay $250/month for this service? To expect to get any sort of scaling capability out of a $8-$30/month shared hosting account is asinine and anyone running more than a small amount of traffic under such circumstances deserves what they get.
Why then if the end service was not capable of sustaining traffic/memory to the same degree that some shared providers are willing to go, would one pay $250/month? I think this is a question the creators of WP Engine genuinely need to answer...
You never, ever, use NFS on something with hundreds of files to load for every page (ie. WordPress).
Every file takes 2-2.5ms to stat/load on NFS (from my tests on WPengine) which seems small but adds up fast.
It means you instantly have 700ms or more of overhead on the SERVER side to render a page (not transmit to the browser, just to render).
You can somewhat get around this by using an opcode cache and turning off file stat but there are still some files that have to be checked like static data read from the disk and turning stat off has other issues when editing code like having to flush the cache.
What you are saying is 1.25ms on your network (5000/4000). That's twice as fast but still significant.
Found this [1], which might be helpful. The linked python program shows really high numbers for me though. Hopefully it's incorrect or I just misunderstand what the numbers mean.
On NFS/NAS it cannot be done locally, because other devices may have had access to the remote filesystem and changed files. So it has to travel the network to make the query. Even if it doesn't have to touch the disk remotely and can fetch it from a remote cache, it's significantly slower than local. Even if it's only 1ms per file, if there are 300 files, that's 300ms (a third of a second) on the SERVER side, before it can start sending output to the internet.
There's all sorts of interesting things we're doing with NFS, along with caching layers, that ensure this is never a bottleneck -- and even provides benefits in other ways.
If WPengine is getting 2+ms response times then something is very wrong (are they using linux backed vanilla linux style NFS storage? ....bad, bad, bad). If not, then they need to investigate their load averages on their filers :P.
We run a relatively small member site (<1000 active members) that operates on WordPress. We tried migrating the site to WPEngine, but could never get the membership plugin working as expected. Over a course of about 2 months we tried everything possible to get it working, and talked with their support on a near daily basis.
Overall, their support team tried to be helpful. They weren't always responsive (sometimes we wouldn't get a response for a day, or we would be waiting on hold for a long time), and we ran into the problem of different support people telling us different things about how their system was setup. Eventually one of the co-founders got involved in the support, but we still were not able to resolve the issue.
I managed to find 2 other people using the plugin that had run into the same problem with us on WPEngine. WPEngine claimed that other customers of theirs used the same membership plugin without any issues. I don't feel they lied, but we clearly weren't alone in this problem.
In the end, we moved to another WordPress host and were able to get up and running in no time. Unfortunately, WPEngine refused to refund us fully for the 2 months we wasted trying to get our site setup because we went past the 60 day money-back period. (We had paid for the 3rd month already, and they gave us a partial refund of that month only.) It was a relatively small amount of money, but after all the trouble we had with them it would have left a much better impression if they would have refunded the entire cost we paid.
I'm sure that WPEngine has thousands of happy customers, and our experience was largely unique, but there are many things I felt they could have done a better job of.
WPEngine provides WordPress hosting services. They are professional and they do it for $250/month (which plainly means that their target audience is professional bloggers). For $250/month, they don't provide consultancy service as why your particular setup couldn't install in their server. This is your responsibility.
Setting up a WordPress blog can be easy (5 minutes) and complicated (Professionals charge up to $350/hour). What you are asking for is someone to move your blog. Fine. You are responsible for that. WPEngine offers you a "6 hours coupon" from another service. That is, they are not responsible for that.
Page.ly are probably doing this to get more clients. This is the wrong thing to do, because their support service can't scale this way. I have been doing this to bootstrap my services and it had more negative consequences than positive ones.
tl;dr: The OP is expecting WP consultancy services for signing up to WPEngine.
... huh?
Edit: are you suggesting that the poor service I got from WPEngine is in fact a cunning conspiracy to boost Page.ly sales?
Because ... well at least temporarily, yeah, I guess it worked perfectly.
"Cui bono?"
-dramatic violin music-
It's a shame that you had to go through all of this trauma to discover this, and there are obvious support failings here, but it does seem that your site is out of the ordinary in some respects, just because of the number of comments on one WP instance.
"Get Over the Paranoia: The story may be apocryphal, but Nordstrom supposedly allowed a customer to return an automobile tire that he insisted he bought at the store. Of course, Nordstrom doesn't sell tires.
"The paranoid would ask, ``But what if everyone who owns a car returned tires to Nordstrom? Nordstrom would go broke!'' But people won't (and don't), and Nordstrom will never be in danger of going bankrupt for accepting returns of goods it doesn't sell.
"Managers are afraid to implement customer-pleasing, revolution-catalyzing policies because they are afraid that too many people will take advantage of these policies, and they'll end up with the equivalent of a store full of tires."
The entire section goes on to talk about employee empowerment, allocentralism (putting yourself into your customer's shoes), examples from other businesses that either do this or don't, and so on; the relevant summary part from the text would probably be, "However, if your competition is asking people to do something suboptimal, then it's creating an opportunity for you."
So, straight up: it's certainly possible to run a business the way that you're describing. In fact, I'd guess that that's how most businesses run: they do as little as possible to make their money and meet -- but not exceed -- their customers' expectations. So you're not wrong as far as that goes.
But you are wrong to assume that they'll have "a ton of noobs" if they treat one particular customer really well. And, you're equally wrong to assume that having "a ton of noobs" is a bad problem to have; from another part of the same section of the book: "For example, a Whirlpool employee taped a news program's interview with Gail, a woman with several children and a full-time job. Whirlpool employees were challenged to provide appliances that would ``take care of Gail.'' In response, they redesigned the stovetop of Whirlpool's CleanTop to be completely flat, without grease traps or dirt-collecting crevices, and they created the Quiet Partner dishwasher, so the noise of the dishwasher wouldn't distract Gail. Viewing chores through Gail's eyes has helped Whirlpool introduce significant product enhancements."
Having "a ton of noobs" would mean a ton of new customers that are paying you money to show you what is wrong with your product or service. They are a great resource to have, if you take advantage of them. If you can identify one problem that 35% of "a ton of noobs" all have in common, and if you can automate the solution to that problem (or make it significantly easier to solve), then you have just achieved differentiation in a crowded market.
Yes, it can be painful in the short term. You might have to hire some more help -- and you might have to streamline some of your other internal process, like your ticket system, which will also pay dividends in the future.
Avoiding pathological customers is certainly good advice for freelancers, consultants, and really small shops. However, it is not necessarily the only good business strategy.
You realize that puts your credibility in the negatives right? Wordpress is infamous for being one of the worst pieces of software ever created. The fact that your software is terrible does not mean a company whose sole purpose is to be experts at making that terrible software run is doing good by failing to make it run.
Get over yourself.
WordPress is a bad piece of software. But there is a lot of money there, and you'll find lots of smart people working on it.
Or as they say in the UK: "Where there's muck, there's brass." http://www.phrases.org.uk/meanings/408900.html
The WP core team doesn't write code with performance, scalability, orthogonal design, good standards or any such thing in mind and you regularly hit gotchas. However... if you sign up to be a WP platform, you're kinda asking for it.
Obligatory performance tips: 1. Cache the heck out of it. CDN, varnish, optcode (APC), query caching on MySQL. 2. As has been noted, file stat'ing is massive. You have to have an exclusive install as close to each serving node as possible, even with optcode caching. 3. Core hack - Modify admin comment searching. Does 5 column wildcard lookups across the entire table. Not scalable. (http://core.trac.wordpress.org/ticket/20487) 4. Core hack - Remove content reallocation on user delete. With large user tables, you lose the ability to delete users (core fix pending: http://core.trac.wordpress.org/ticket/19867) 5. Clean and optimize your DB regularly - Old post revisions are a huge gain 6. Add indexes intelligently - WP doesn't include all the indexes you need 7. Horizontally scale your DB (whether through MySQL clustering, or HyperDB + DBs with roles) 8. Tune everything. I've found that NGINX/Apache and Percona/MySQL make little difference, but setting the right config for each daemon makes huge difference.
How fast is... Apple.com? (tl;dr: not very); Wired's Review of the Microsoft Surface (tl;dr: pretty interesting).
"Apple.com takes 25 minutes to load" "Wired says Microsoft Surface is Quite the Stunner"
The original author may want to use that title but that's not to say the person who posts it to HN has to.
Oops.
Go with Rackspace. They're expensive compared to budget hosts, but they seem to have an army of sysadmins available 24/7/365 that never say "that's your problem" / "not in scope".
PS: that redirect bug is probably a mismatch of your site_url setting in either wp_blogs, the wp_options table of your primary blog, or the wp-config.php. If the domain set in any of those 3 places doesn't match you get that endless redirect loop. Could also be htaccess sending you to www or no-www, and your config is set the other way. Its an annoyingly common issue when moving MU sites, and they should have been able to fix it.
Since they 'are' WordPress they can make pretty much anything that does not modify core, work. It's not for everyone, but for the right sites (high traffic/high availability/generating revenue) it's fantastic.
Until I came to this sentence, I was coming to the conclusion that CMSes fall down fairly quickly once you add in large data, large traffic, or complex installs. For the life of me, I can't see why people continue to use CMSes for large projects anymore, when they could potentially code something up faster in Django/Symfony/Rails and, here's the kicker, know the codebase inside and out.
My client has simple marketing (read: 99% static) websites that are built on Drupal. Sounds good at first, and it almost makes sense... and then you get to the non-functional requirement that it needs to withstand a marketing event that can bring 2k+ concurrent users for over 24 hours. Then you watch Drupal meltdown the server.
Email me if you're interested in reading the proposal, I have it lying around somewhere.
I can see a lot of scenarios where you start with a CMS because it does a good job of doing what you need. Why reinvent the wheel?
Scaling issues is going to occur in all code, even code you write and know inside and out. It just means you become entirely responsible for debugging and re-architecting to improve its performance.
@Drupal specific comment, you could easily throw nginx as a reverse proxy with a cache in front if everything is static and be done with it.
I went through 3 or 4 different providers, and the only place that i was really happy with setup, infrastructure and support was WPEngine.
As some of the other guys have stated regards the caching and their suitability to vanilla installs, i think that is most likely the case, and yes mine was a pretty vanilla install in the grand scheme of things, but i'd still choose them over anyone else.
Ok, so first and foremost I am incredibly sorry, Jacques, for the experience you had. I can say with confidence that your experience with WP Engine is out of the ordinary, but no excuses. I’m writing this as a way of owning up to mistakes that were made on behalf of our company, as well as transparently engage the discussion here. There's a couple of points I'd like to make, if I may...
We're in the process of moving to complete 24/7 support right now (we do have emergency 'site down' 24/7 cover) - as any entrepreneur on HN will tell you, getting your startup to take off is hard and scaling the human aspect is perhaps even harder. With mostly US-based business customers it wasn't as much of an issue to begin with but we're now addressing that.
Obviously if we'd had that 24/7 support in place we could have addressed this more immediately.
However, one of the reasons that your site experienced problems was that there were some aspects of the site development that simply didn't scale. Like any scale-orientated PaaS, we can provide the infrastructure to enable you to scale but your code needs to work in partnership to fully achieve that.
As Sean, my head SysAdmin wrote on a ticket to you, some of your pages were performing a sort on 188590 rows in memory each time. That just doesn't scale. From your post, I'm guessing this is the issue that also occurred when you tried Pagely. While it's awesome that the good folks at Pagely were able to work with you a little to try to address your problem, like most PaaS providers (eg AWS, Heroku) we don't provide consultancy services - there is an amazing WordPress community of consultants and dev shops out there and its just not what we want to get into. Just like Heroku won't fix your Ruby code.
We have many, many clients who are very happy with WP Engine but as a growing business there will always be customers who find that the service isn't right for them, and even occasionally have a bad experience.
So lets try to make this right. I can see that we've already refunded you all the hosting fees you paid for your account, and I'll also make sure we pick up the $500 migration bill that you incurred. If there is anything else we can do please let me know.
I will also take time once I've finished my coffee and out of my PJ's to analyze your case further and see what else we can do internally to ensure this doesn't happen again.
Bests,
Ben, co-founder WP Engine
FYI, the problem is with the Recent Comments widget code that ships in the Wordpress base install. Page.ly had the same database connection dropping issues you did and I presume for the same reason. (edit: By the way, use the db-error.php facility to put up nicer error pages. It's also a really obvious point to insert some monitoring logic -- I think customers would like it if you noticed their problems before they do).
The core issue, as I said elsewhere in this thread, is that unless you partition the underlying wp_comment tables, performance becomes steadily worse as the total number of comments rises.
The only reason that this was never a problem for me on my own servers is that I am loltastically over-provisioned for my requirements. I just never saw it.
You will see it more in future. Like I said: this is not a badly behaved 3rd-party plugin. This is mainline code. Your customers can reasonably expect that you can run Wordpress out of the box without degradation.
None of changes the fact that my original migration experience was a total disaster. I know you're experiencing growing pains.
Well here's one of them. I hope it serves you and your future customers well.
Yet in my pinboard I have this bookmarked: http://pento.net/2011/04/28/partitioning-the-wordpress-comme...
Yup, I had to partition the comments table.
I've been watching this too: http://wordpress.org/extend/plugins/postgresql-for-wordpress...
I'm willing to accept fractionally slower page loads, as for the most part they're cached heavily post-generation. I'd feel far more comfortable partitioning in Postgres not least because I'm far more familiar with it.
https://twitter.com/asmartbear/status/261103840231825409
https://twitter.com/asmartbear/status/261103530855784448
Ben, I really appreciate you taking the time to post this here and demonstrate that you take this seriously, but your cofounder is currently publicly ridiculing customers who have paid good money and been dissatisfied with your service.
As a current customer of WPEngine (who has had issues), nothing makes me want to bail on you guys as much as those two tweets. From the CEO.
Completely disgusting.
EDIT: This is more what I expected to see from Jason, though even this has a weird tone: http://news.ycombinator.com/item?id=4086553
@billmcneely yup, there's always someone bitter and hateful. Sometimes they even have a point, but choose to make it in a destructive way.
@P_Doubleya @gozmike thanks guys. That's also why I ignore HN. There's always haters.
Of course you may disagree with the tone, or the content, but disgusting is a little strong isn't it? Can we dial down the hyperbole a little?
I'll grant you that "ridiculing" is probably not the right word...maybe dismissive or condescending would have been more accurate.
And yes, I do find it disgusting that he's publicly behaving this way towards a customer.
It's fair to criticise those tweets and I don't feel they are appropriate really, but I don't think they should be classified as disgusting - misguided and inappropriate at this particular time, perhaps, but not disgusting. Isn't it better to deal with situations like this without escalating emotions further and degenerating into both sides insulting each other?
The completely different service refund wouldn't be necessary if the not different service had been up to expectation in the first instance.
I feel absolutely justified in being refunded that amount by WPEngine.
All I want to see from the CEO in a situation like this is: "I'm sorry. We fucked up. Here's how we'll make it right, and here's how we'll make sure it doesn't happen again."
And if you genuinely can't say that because you don't think the customer is right and you're not willing to do anything for them, just keep your mouth shut.
I think a great heuristic here is this: your customers can be publicly negative, but you cannot.
If you publicly flame a customer then, no matter how deserving the customer was, some future potential customers will look at that and say "That could be me." Most businesses, I think, can't afford to risk that.
There are some notable counterexamples, in my opinion they are the exceptions which prove the rule, i.e. when the business wants to fire the customer and make an example of him/her for others.
The best example of this is the Drafthouse movie theater in Austin, who created a great PSA wherein they ridicule a customer for insisting she had the right to use her cell phone during a movie: http://adland.tv/content/drafthouse-movie-theatre-makes-keep...
I agree 100%, the tweets are not helpful, but I disagree with the "disgusting" description and felt it was a little over the top.
Would you consider it reasonable to be the subject of the manager's tweet, or open letter or any other kind of public message calling you someone who cannot be pleased, who has an agenda against the store, is disposed toward bitterness, and is communicating destructively.
How is that not disrespectful?
Edit: minor
I actively ignore Twitter for the same reasons some might avoid HN, but that doesn't mean that there aren't valid and intelligent conversations to be had there, particularly if you have to deal with customer service and PR.
I'm neither justifying nor criticizing what he said, just let's be careful not to burn someone or something based on one mistake that we all can one day find repeating.
His clarification tweets:
https://twitter.com/asmartbear/status/261189038696960002
https://twitter.com/asmartbear/status/261188913991913472
I'm sure he'll be more careful on public tweets next time, but I think jumping to conclusions on his company etc is going too far.
He made a mistake, obviously seems he learned from it, and we all learned a nice little lesson in customer service, public relations and social web dynamics.
Ben, I don't know Jacques and don't pretend to speak for him, but I gather from his post that the scaling issue, and you not being able to accommodate him, would have been no big deal if he had not already been through the nightmare of migrating with bad documentation and support not handling that properly.
I think cascading frustration is what got him.
I'm incredibly proud of the business the team has built, and know we are still an excellent place to host WordPress. But yes, incidents like this give us opportunity to revaluate aspects like our documentation (and support hours) to double down and make things even better one we resolve them.
Since our customer was looking for a more agile development & deployment workflow I personally recommend them switching from us to WPengine based only on Jason reputation and your technical expertise.
Almost a month has passed since they signed in for the service, $500 in fees and ~ $700 on WPmovers afterwards, they keep hosting with us and couldn't arrange a 0 downtime migration at all, neither provide a fixed quote for the migration.
Our case right know is being handle by Sergio, maybe you can talk with him and our customer to find a happy ending?
I've always been an advocate of WPengine and big fan, but after our bad experience and seeing others having similar issues too, I'm probably not recommending WPengine in a while..
Regards
M
As Sean, my head SysAdmin wrote on a ticket to you, some of your pages were performing a sort on 188590 rows in memory each time. That just doesn't scale.
From a customer service perspective, this comment leaves a really bad taste in my mouth.
You guys say right on your home page that you're "infinitely scalable." And then if you click through to http://wpengine.com/scale-to-millions-of-hits-a-day-or-hour/ you read that WPEngine features "secret sauce" made of "one of the most scalable WordPress architectures on Earth" that makes you "faster and more scalable all the time." The (strong) implication being that any WordPress configuration will perform better on WPEngine than on an alternative hosting platform.
So if someone comes to you with a WordPress configuration that you can't make "faster and more scalable" by running it on your platform, isn't The Right Thing To Do to say "oops, our bad, sorry we misled you" rather than blaming the customer for having a bad WordPress configuration? Because that's what you're doing here -- blaming the customer.
In fact your comment basically boils down to three parts, only one of which is reassuring:
* Paragraphs 1-4 translate to "we're a growing startup and growing customer support is hard," which frankly if I were a customer I wouldn't care about in the least. Scaling your business is your problem, not my problem. We know it's hard, but it's like Tom Hanks said in A League of Their Own (http://www.youtube.com/watch?v=tPqYnC-SW5w) -- there's no crying in business.
* Paragraphs 5-7 translate to "your web site sucks, no wonder we couldn't scale it, nobody could," which is the blaming the customer part. I don't doubt that that is true (WP can be a real hog, especially if it's hacked up), but if you're advertising yourself as The People Who Can Scale WordPress, you shouldn't be surprised when people start coming to you with funky, hacked-up, hard-to-scale WordPress sites. Who else would someone with a site that can't scale go to but the scaling experts? When that happens it's OK to step back and say "we can't scale this," but it's not their fault for taking your marketing materials at their word.
* It's not till Paragraph 8 that we get to the "let's try to make this right" part, which is where this comment should have started.
I don't say this to be a pain or to make what I'm sure is already a difficult day worse. I'm saying it because I like what you guys are trying to do, and responses like this hurt you as much (if not more) as they help.
I've had to deal with situations like this myself, and no matter how tempting it is to blame the customer -- even when the customer is manifestly at fault -- it never, ever, ever improves the situation. It just makes things worse, first by giving the angry customer something new to be angry about, and second by deflecting you from the more important question of what you can do to prevent the situation from happening again. In this case, for instance, better screening of incoming customers to ensure their sites really can be scaled on the WPEngine platform would filter out problems like this before they become problems. (IIRC you've already started doing this, by blacklisting a small set of known-bad plugins. Maybe that list needs to be beefed up?) But time spent sniping at customers is time not spent making those improvements happen.
No, there's no inference or implication that any (ie regardless of how its built) WordPress configuration will scale on WP Engine. As I wrote in my prior comment, scalability is a partnership between infrastructure and code. That's the case irregardless of framework and PaaS provider.
We like car analogies at WP Engine - if I sell you a racing car and you're only a moderately good driver then you're going to crash when you approach the corner at 200mph. There's little I can do about that other than teach you to drive, and I'm in the business of building the cars. There are plenty of folks who can help you improve your race-craft out there.
To say that we've missled someone is an unfair assertion.
If I analyze the suggestions you're making in your comment here...
* Jacques complained about lack of 24/7 support, but I shouldn't have tried to reply and explain the reason for that because 'there's no crying in business'. ?????
* The WP Engine website shouldn't mention scaling WordPress because it's not technically possible to scale all WordPress code regardless of how it's written. (What should our site say? What should any scale-orientated PaaS website say given the same issues apply to anyone in this space)
* WP Engine should vet and screen it's customers so that they must prove their code can scale before we sell them an account? ?????
I'm sorry, but if that's the advice you're offering then, with respect, I don't agree.
Edit: and Wordpress is a monster. But that's why I wanted to pay you guys in the first place.
That being said...
No, there's no inference or implication that any WordPress configuration will scale on WP Engine.
I would humbly suggest that your entire Web site is a refutation of this assertion. Even moving beyond the copy on the home page to the fine-print "Common Questions" section of http://wpengine.com/pricing/ , there's no disclaimer or other "are there WordPress configurations that WPEngine can't scale?" fine print.
My point here isn't that you shouldn't promise scaling, it's that it's not hard to see how a mismatch could arise between your expectations (scaling is hard, customer code can make it harder) and customers' expectations (scaling is easy, WPEngine has figured it out).
Jacques complained about lack of 24/7 support, but I shouldn't have tried to reply and explain the reason for that because 'there's no crying in business'. ?????
Yes. If a customer complains about a lack of 24/7 support, you say "I'm sorry, but we're not in a position to provide that level of support at this time." And nothing else. The reason why you're not in a position to provide it is irrelevant to the customer, and risks sidetracking the conversation into a blind alley about the reasons.
It also feels a little bit like pleading for understanding -- "cut us some slack, this stuff is hard!" -- which is off-putting. Of course it's hard, that's why people pay you to do it for them.
WP Engine should vet and screen it's customers so that they must prove their code can scale before we sell them an account? ?????
As I said, you already do some vetting by disabling plugins you know to be performance-killers. Look at this list: http://support.wpengine.com/disallowed-plugins/ , there's an entire category called "CPU/MySQL Thrashing Plugins."
I'm not suggesting that you need to give your customers a full CIA background check before bringing them on board, but you've already moved beyond just taking whatever code customers throw at you. If there's other things that need to be disallowed to ensure that WP sites work on your platform, the thing to do is to find out what those things are and add them to the blacklist.
Again, just my $0.02, IMHO, YMMV, IANAL, etc.
Use some common sense.
"infinite scaling" means _exactly_ that. If you don't want people to use accepted dictionary definitions of words like "infinite", you should not use them in your marketing.
</devil's advocate>
A few telcos here in Australia have recently lost court cases about their use of the word "infinite" in their cellphone plans/marketing, and then cutting off or charging extra to customers who "used too much". The court, rightly in my opinion, said that the meaning of the word "infinite" is clear and well known, and any claim that it included "limited, of course, by common sense" is bogus. Be careful of letting your marketing team write too much hyperbole - occasionally you'll be held legally accountable for it.
I suspect if the amount in dispute had been an order of magnitude or two larger, Jacques may well have had a strong enough case to find a no-win/no-fee lawyer to go into bat for him... I know I'd prefer not to have to explain to a judge how those first two boxes in the first column on the WPEngine homepage didn't cover all of Jacques complaints.
I can create an infinitely scalable server and you can come in and drop a sledge hammer on it...does that mean I shouldn't use the word.
You have to understand the layperson won't know anything about tables etc and stating "infinitely scale" is a HUGE selling point for potential customers. If they wanted to be truthful they wouldn't say that, or they would say we CAN provide infinite scaling IF the following conditions are met.
The way I see it, you are not selling the car, but providing a driver so that I don't have to worry about how it should be driven.
I should sit in the back and relax while you make sure that my car is driving safely and comfortably.
Maybe your message isn't clear enough if you think that your product is the car. From your website:
QUIT WORRYING AND LET US RUN WORDPRESS FOR YOU
I read:
QUIT WORRYING AND LET US DRIVE YOU CAR FOR YOU
It seems to me wordpress.com is the "we supply the racing car and driver" solution provider. You don't have to worry about how it gets driven, but you also have only limited say in what car you get and which lines the driver takes through corners.
In my version, WPEngine are providing the race track - to which you bring your own car and driver. They go to a bit of effort to improve the safety, mostly by ensuring other track users don't do too much to affect your performance, but you're free to bring your own F1 car, MotoGP motorcycle, Nascar, or go kart; and run at whatever pace you think is appropriate.
Seems to me the OP's problem was that he wanted to run a top fuel drag rail. From his perspective, he made assumptions about what "a race track" meant that didn't include "decreasing radius downhill hairpins". Meanwhile WPEngine's website/marketing didn't anticipate that class of user, and didn't think they needed to point out that their track isn't an ANDRA approved 1440' long straight with a mile or two of slightly uphill runoff at the end.
With my sympathetic hat on, I can easily understand why both parties assumptions were sensible to them when they made them.
With my cynical hat on, it's pretty obvious that you could install a Wordpress theme or plugin on anybodies hosting platform with an O(m^n) or O(n!) function it it somewhere, and no magical cloud autoscaling pixiedust is going to solve your problem. At the same time, the expectation that a company claiming "Insanely Fast. Infinitely Scalable." and "At WP Engine, there’s no “first level” of support–our entire staff are WordPress experts, so you never hear “We don’t know how to do that.”" on their homepage should be able to do a better job keeping a Wordpress site up than some time-limited non-technical guy self-hosting on Linode.
FWIW, it's clear that you _did_ mislead Jacques.
Whether the assumptions he made were valid or not (and frankly, based on the content of your homepage, it's hard not to have sympathy for his point of view), he clearly expected you guys would do a better job hosting Wordpress that he could do himself on his own Linode VPS. He also clearly expected that if there were problems, your "Expert Wordpress Support" would provide the solutions. Instead he found your support guys effectively saying "we don't know how to do that" and flicking him on to WebSiteMovers, who _also_ end up saying "we don't know how to do that" and charging him $500 for failing.
So I think you did let Jacques down, and I think he has every reason to feel missled.
Constructive advice:
You should probably acknowledge, at least prominently enough that anyone researching your service with a view to becoming a paying customer will find it, that there are possible pathological cases and configurations which WPEngine's technology/infrastructure/service will not solve. If you don't want to feature this on the homepage (and I can understand you not wanting too), I'd suggest a note near the top of the Disallowed Plugins page - and probably re-wording that page to make clear that avoiding everything on the "complete list" of disallowed plugins doesn't guarantee a successful deployment, and also that it's possible for themes with certain code patterns in them to cause a WPEngine deployment to fail to perform as advertised.
Secondly (and I suspect you've already got this one in hand), you need to train your support staff to recognise more quickly installations with pathological performance things going on, and apologise and extricate yourselves and inform the customer as early in the process as practical. I don't know what the tone/implication in Seans message about the ~188k row sort was, but from reading your defence here I can't help but think there was at least a bit of "you're doing it wrong!" - which even when it's true, it's not the wording you want to feed an unhappy customer.
(Note: this is written as a happy WPE user, with some curiosity about exactly what went wrong, and why some guy with a Linode VPS can, in some cases, keep a Wordpress site up better then WPE. I'm suspecting he's got define('WP_MEMORY_LIMIT', '2048M'); or something similar covering up some quickly hacked together "it works for me" not-very-well-though-out-code - which you can get away with on your own hardware/VPS, but not generally on shared hosting...)
There's a big difference between being infinitely scalable, and being infinitely foolproof. If your PaaS exposes enough power to the user, they'll find a way to shoot themselves in the foot occasionally.
You guys say right on your home page that you're "infinitely scalable." [...] The (strong) implication being that any WordPress configuration will perform better on WPEngine than on an alternative hosting platform."
I agree with your general point that, from the standpoint of the entrepreneur, "the customer is always right" should be words to live by. On the other hand, in the paas industry, it should be axiomatic, i.e., understood by all competent parties, that horizontal scaling (the "infinitely scalable" bit) manifestly will NOT address problematic architectures, n+1 queries, full table scans, poorly/non-indexed schemas, etc.
This, to me, is not blaming the customer, it's laying out in reasonable terms assumptions any competent developer or software entrepreneur should understand long before they employ a paas.
Your larger point, though, is valid: even after bracketing the given assumptions of what the responsibility of a paas truly is, better communication up front could temper expectations as to what "infinitely scalable" (a bold phrase, to be sure) really means to the individual customer. It doesn't mean a paas can fix a application that locks the datbase for a full second per query, etc. It does likely mean (I'm assuming here, I have no experience with WPEngine) spinning up new instances and handling increased loads should be Heroku-simple.
In this case, I think WPEngine is taking the hit for what they should (bad communication on the support side) and explaining in real terms what paas's can and can't do. There's a difference between "blaming the customer" and simply being honest about what the bottom-line issue is. NOT addressing this side of it may have long-term implications if another customer comes along with similar expectations. Being clear about what the client's responsibility is doesn't necessarily equate to "blame."
The problem with this is that the original post doesn't say "I have a site that was crashing and locking up all the time, so I took it to WPEngine." It says "I have a site that was running fine on a VPS at Linode, but I got tired of the hassle of managing the VPS, so I took it to WPEngine."
The original post is written by the complainant, of course, so it's possible it's subjective or slanted. But if it's correct on this point, I don't think it's unreasonable for a competent party -- in this case, a person with a WordPress site that runs acceptably on a commodity VPS -- to expect that it would run acceptably on WPEngine as well. Presumably performance killers like the ones you cite would be just as bad on the VPS -- if not worse, since it's unlikely that a VPS' operator will be able to tune MySQL, PHP, Apache, etc. to the degree that an expert at a place like WPEngine would.
The Most WordPress Experts Per 1000 Customers…
Great support? Many hosting companies claim this. But when you’re hosting WordPress, you need specific support. If a plugin breaks, or a theme file doesn’t work, and you’re hosted with a typical hosting company, they’ll throw up their hands and say “Not responsible! We just provide the server and the bandwidth, buddy.”
Not so at WP Engine. Our staff consists entirely of WordPress experts, so we can help you track down and troubleshoot WordPress issues.
Had to move our highest traffic site back to GoDaddy (which makes me cringe, don't get me wrong). Doing the same with our other two sites.
That said, I believe in what you guys are doing, think there's space in the market for someone like WPEngine ... it's just that, in my experience, you guys are executing poorly and dropping the ball quite often.
Which is fine, but then you website says this:
- http://wpengine.com/wordpress-experts/
Not so at WP Engine. Our staff consists entirely of WordPress experts, so we can help you track down and troubleshoot WordPress issues.
Which is totally misleading, and exactly opposite of what you say. I wouldn't blame the customer for dissing.
Edit: Quote formatting
You quote 'we can help you track down and troubleshoot WordPress issues.' - that's exactly what the support team helps customers with all day long.
But we don't get into consultancy around helping to build custom code, etc.
There's always going to be a point where a hosting provider will not be able to support you further. Many dedicated providers and VPS providers will ensure that the machine is powered up and has connectivity and leave you at that. What we're trying to communicate is we go a lot further into supporting your use and configuration of WordPress - but not into consultancy.
I know Jacques didn't mention that the problem was with the stock installed-with-Wordpress Recent Comments widget until a couple of hour after you posted that comment, but _surely_ someone at WPEngine knew already that this wasn't a "custom code" issue?
From what I'm reading (and please correct me if this is wrong) a stock WP install with no plugins or custom themes, but with several hundred thousand rows in the comments table and the Recent Comments widget running - will crash hard on WPEngine (while running just fine on a "loltastically over provisioned Linode VPS).
In my view - that's a "Wordpress support issue" suitible for diagnosis by a "Wordpress expert", and a long way from "consultancy" for "building custom code".
It's true there needs to be a limit to what customers expect from hosting companies, but like it or not, you've set the expectation bar _very_ high with the marketing copy on your homepage, and you've got a wonderful social media reputation based on those expectations. At some stage, you need to suck it up and dedicate the time/expense to live up to that reputation, or inevitably social media will bring your reputation back down in line with your business behaviour/policies - and the wider the gap between the expectations you've set and the service you deliver, the more ammunition you're handing over to "the haters" to shoot you with.
I can understand that the customer is frustrated with having to change his code to move from a VPS environment to a cloud environment, but that is the unfortunate truth when scaling. Either you 1) get closer to the real hardware and throw resources at it (more ram, vps, or dedicated machine) or 2) change your code to be horizontally scalable on light cloud instances.
Maybe this is chance to create a help-page that explains to users what resource limits they should stay below (memory, cpu, queries) to handle peaks in traffic and how to identify problem points (bad SQL queries, hungry plugins).
...if I were in your place I'd drop the "solving this while yawning in my PJs" relaxed attitude and prepare to bring a sleeping bag to work and stack a fridge full of red-bull (or whatever makes you "fly") ...or try and hire someone like Ledorf ...'cause all the other "PHP gurus" are at Facebook now ...tough luck. I'd hate to be in your place, but no matter how much I hate the "PHP ecosystem" I hope you guys manage to make this work
A friend of mine once said, "a customer will defend his bad decision to the death if you point it out to him."
I've been running a site about the same size as the OP's (500k-ish visitors a month) with WPEngine for nearly a year now, and so far I'm very impressed. It's smoother and faster than it was before, we've only had one outage of more than an hour (and they responded by crediting me with a full month's hosting!), and I've found their support to be absolutely top-notch.
I wrote a review of their service after a month - which I don't think I ever submitted to HN, d'oh - and all the points I make in it still stand: http://www.mmomeltingpot.com/2012/03/wpengine-review-after-1... . My key point there, which I stand by, was that moving to WPEngine saved me money over an apparently cheaper self-hosted solution, because I no longer had to spend 2-3 days a month faffing around with server problems. (I believe Patio11 had a similar experience.)
(Note - there's an affiliate link in there, because I find that with money I can buy goods and services, but I added that after writing the review.)
Just recently I've had an extremely helpful dialog with one of their support engineers which ended up troubleshooting a problem I didn't even know my blog had - which subsequently boosted my available ad inventory by about 25%.
Overall, WPEngine aren't perfect - and I'm still checking out other options from time to time - but I've found their support and general solid hosting more than worth the admittedly pretty significant cost.
We went live on a WebbyNode VPS, thought we had configured our system well and suddenly we hit the top spot on HN. The server was on fire and Jason personally showed up and worked on migrating our site over.
Your configuration may have posed problems, you may have had what seems like a shitty support experience, but this public bashing doesn't help anyone in the startup community and reads as if it's motivated by emotions rather than a real problem-focussed attitude.
Give these guys a break and work with them one on one to find a solution to your problems. I'm SURE they'll move mountains to make you happy because that's in the team's DNA.
Weeks.
They did not move mountains. It was a parade of apathy. As soon as they could flick pass me to WebSiteMovers, they did.
Your positive experience is every bit an emotional one as my negative experience was.
It really sucks that you went through this. I'm just suggesting that you refrain from the public flogging and find another solution. I'm not affiliated with these guys other than as a client and I can tell you that I respect what they bring to the market.
Especially given the content and tenor of their marketing.
If you've spent more than a day in any online community you should have already learned that this, although it may not be preferable for the company at fault, is the fastest way to get something fixed or something changed.
It'd be irresponsible for Jacques to stand by silently and suck up all the time and money he wasted days after there was a great PR piece for WPEngine on the front-page of this site because it would lead to others having his issue which he could help prevent.
This is a community. It's not an advertisement source. People vote on things they find interesting and discuss them, it's not a place to only post positive things about up and coming companies and hide all sources of negative news.
You're advocating revenge. I'd just like discussions to be discussions. Not attacks.
The blog post read (to me at least) like an attack on a company with some obvious growing pains.
There are ways to write a complaint that doesn't involve pointing out Cohen's self congratulation. The way the post is ended wreaks of jealous hatred. That is not the way you constructively criticize.
It seems like he's spent an extended amount of time complaining to internal channels, and he just needed to let other people know about his experience of not receiving the promised levels of service.
Wait, what? Where? I saw nothing in that entire diatribe that sounded like a call to arms.
Where else am I going to read this?
Give these guys a break and work with them one on one to find a solution to your problems.
It took a 10 minute phone call. I knew the names of everyone on my case and they kept me updated through the process.
I'm SURE there was a big fuck up in the support process for the OP. I just feel that there are other ways to skin a cat, particularly in our community where we should be working hard to find constructive solutions to things like this instead of creating a cloud of negativity around young businesses.
I am in Perth, Australia. This means I need to stay up until 10pm in order to speak to them.
Like I said, that part is on me for not finding out their hours in advance.
Here is a great example from a radio personality I used to listen to:
http://www.kalzumeus.com/2012/02/09/why-i-dont-host-my-own-b...
I haven't tried it but it seems to be a good experience for some people.
I read it again before I signed up with them and googled around for reviews. Everyone was very positive.
That being said, once you have the sort of requirements that the original poster seems to have, I don't know if any of those companies would be suitable…
I used to host Club Troppo there. When it got slow they basically shrugged and said ... it's shared hosting, what did I expect for $8/month?
I had to migrate my WordPress off Dreamhost shared hosting as it was too slow and blowing their CPU limits. That was around 2006 and I'm sure a lot has changed.
The only reason I'm thinking about switching away is that their control panel is a little messy, and they don't support Git deployment out of the box (although you can activate it via ssh, which is what I did).
It also feels faster than PHPFog for some reason, especially the WordPress admin. So overall, I'd say they're a pretty good option for budget hosting.
If you're not going to be doing a lot of dev work and making a lot of commits, I'd recommend.
I'm not sure if it's true, but based on the poster's description of WPEngine's customer support experience, it sounds like their customer service is the typical poorly structured (from the customer's point of view) customer service experience I've encountered and/or witnessed many times from companies both large and small.
Some startups don't provide phone support. It's nice to see WPEngine does. Sometimes you just have to talk with someone to beat it into their heads that you have a complicated problem. They can't email email you back a link to some topic-related doc page and move on. They're on the phone with you and they have to make meaningful progress before hanging up.
In the case of email support, 95% of the tickets are answered with a link to a doc page and that's the end of the case. The metrics are excellent and every pats themselves on the back. 95%! And the volume was soooo high! Acknowledging positives is necessary, but measuring failure requires different metrics and 95% success does not necessarily imply 5% failure because each one uses different metrics. You think the the metrics on their dashboard will take into account this thread, for example?
The system breaks down when the customer has a more serious problem. It takes support a few emails back and forth to realize it. The case is handled by random folks based on availability, further degrading the experience. The person who'll pick up your latest reply to the thread isn't going to thoroughly read the whole thread, and by this point you've interacted with five support staff already. Finally, they realize it's outside their control and pass it off to Engineering, where it languishes for a month. Creating a system where the engineer is not allowed to reply directly to the customer but instead has to reply to support who then replies to the customer only removes the stress off the engineer, so they can care less and not solve the problem. Meanwhile, the support staff learn to drop lines like "Sorry for the delay. We appreciate your patience. The engineers are still looking into it." without any second thoughts.
My case was not an edge case. WPEngine explicitly mention that they support Multisite, which has been part of the Wordpress mainline since 3.0. I haven't modified it at all.
Otherwise, I agree with your post. Support is hard.
I don't doubt this was a shit experience, it certainly sounds incredibly infuriating.
Jason & his team work super hard, and have thousands of happy customers, so they're doing lots right. Even the greatest of companies can screw things up now and then, especially in non-straight-forward situations. I'm sure he'll put it right.
WAT! Just add the domain to your hosts file with the WPEngine IP and test the site first, it's crazy to change your DNS without checking the site works first.
I've learnt this lesson well.
If you can perform the comment sort in memory without hassle, what is is about the size of RAM provisioned by wpengine that makes this impossible?
My opinion, but for for $3000 per year, you should have been provisioned with as much ram as a linode VPS.
In case you missed it up-thread:
https://news.ycombinator.com/item?id=4695810
two 512mb linodes, one for nginx/php, one for mysql.
> It’s a quiet backwater. Collectively the sites would have maybe 20k unique visitors a day. 50k is a big day hereabouts.
I'm not quite sure why this blog is such a huge problem to host. It appears that the author has some customizations (plugins..) that are really slow. In fact, I would suggest to the author to hire a good developer, fix those core issues (say slow SQL queries, etc.). It might cost him a few thousand dollars upfront, but he can continue running the site on $20-$60/month hosting.
Paying $250/month for that kind of traffic sounds over the top to me.
I know that Wordpress can be slow but it can also be quick. Why not route your traffic through cloudfront ? Why not leverage full page caching for anonymous users ? These are things worth looking at IMHO :)
The slowest piece of code on the network today is the Recent Comments Widget. That's not a plugin, that's right there in the base install.
I worked this out myself, based on preliminary detective work by Page.ly. And then Page.ly went the rest of the way to ask a Wordpress core contributor for their opinion. The key issue is that this widget performs a query on the wp_comments table which becomes slower and slower as the number of comments rises. The only real fixes are either to move to Disqus (my bloggers said "no") or partition the table (Page.ly politely said that this was outside the scope of anything under their enterprise plans).
> Why not route your traffic through cloudfront ?
I do.
> Why not leverage full page caching for anonymous users ?
That too.
I also have memcached for the object cache, I use a tuned copy of PerconaDB and nginx is configured to serve static files and supercache-generated whole pages without ever touching PHP.
The blog post you read was served from a blog on that very blog network, on Linode servers in Tokyo that I once again personally managing.
How long does the query take?
How many rows do you have?
Sorting that into a temp table, and caching that in memory would be really fast on a server with reasonable RAM and proper indexes.
I dont' see why a selective read based on primary key indexes is taking so long. How many rows are we talking about here ? I'm sure it's under a billion rows - right ? Partitioning it may help but I think there's something significantly wrong with the way the lookup is going through.
The recent comments widget is probably doing things the wrong way around. Is it caching the returned values ? Does it matter if the recent comments widget is based on data that is.. say 2 minutes old ?
I really don't agree with your performance bottlenecks. I run several sites much larger (at-least 20x traffic per day) and granted that they do not all run Wordpress, I know from experience that your system should not be grinding to a halt with just 50k daily uniques.
Even imagining that the comments lookup is like "get all comments for post ID, 2966 --" - that can't possibly be slow enough to destroy your entire server setup. Maybe you just need more RAM ?
Get a dedicated box and get someone to optimize it. If your entire database can be in RAM (Get a box with 16GB of RAM ?), that should work well. That won't cost you $250/month even.
EDIT: I see from another post: As Sean, my head SysAdmin wrote on a ticket to you, some of your pages were performing a sort on 188590 rows in memory each time.
Was that the recent comments widget doing that ? Well, that can be fixed with some clever caching or even a clever fetch technique!
Is that necessary for every load ? Do you have a particular blog post with that many comments ? The right way to go about that would be to pick out comments for a particular post and then sort them - query-wise. Again, your WP install probably needs some (much needed) tuning.
I understand your desire to have someone else worry about your servers, but someone also (apparently) has to worry about your code, and as I understand right now - that person is you. Well, you could definitely make some code level improvements and save on the 'server management' department.
I only see one query in there: $comments = $wpdb->get_results("SELECT $wpdb->comments.* FROM $wpdb->comments JOIN $wpdb->posts ON $wpdb->posts.ID = $wpdb->comments.comment_post_ID WHERE comment_approved = '1' AND post_status = 'publish' ORDER BY comment_date_gmt DESC LIMIT 150");
I'm not sure why they are fetching 150 (why would you use that many?) but this query could be slow if you have many posts and many comments. Also, the goofy thing is that they are wiping out the cache every time and regenerating it instead of (say) incrementally adding to it.
I'd re-write it with an exists, which may help performance instead of a join (I've seen MySQL be pretty dumb with joins when you basically want an exists):
SELECT whatever FROM comments WHERE EXISTS( SELECT * FROM posts WHERE ... ) AND comment_approved = 1 ORDER BY LIMIT xx
In this case, MySQL may be having a problem where it can't read a multi-column index on post_id and comment_date_gmt DESC (I'm not sure if you can specify ordering in MySQL indexes, been a while), thus it's basically doing an in-memory sort of every approved comment (on a published post, which is likely all of them) in the system, which in his case was about ~200K. This could definitely take several seconds if there isn't enough memory to do it all at once (lots of swapping).
Personally, I'd try and modify the logic to be more conducive to indexing - do you really care if a comment was on a post that you made 10 years ago? MySQL also (IIRC) doesn't have bitmap indexes, so that "comment_approved" thing may or may not invalidate an index usage unless you add it to the index.
I'd probably do something like: the "n" (where n is < 150) most recent comments to a post made within the last n number of days, so you can really cut down on the number of rows you need to sort.
But as I said elsewhere, the performance thing is just the straw that broke the camel's back.
Reading about this, I get concerned when people conflate "performance" and "scalability". Performance is about time-based metrics such a requests-per-second. Scalability is about the question of whether I can add more hardware proportionately to my request load and continue to perform. WPEngine's marketing claims are about scalability, not performance.
This guy's WPEngine code was apparently doing a 189k row sort when serving certain requests, which is not going to perform well on any hosting platform, including DIY.
WordPress.com is an SaaS for blogging like Tumblr or Posterous. It happens to be built on a custom, locked down install of the WordPress multi-site software. It's not a website hosting company.
Edit: and oh yes, 2.5Gb of media.
The big kicker is that my migrations can only occur by SQL. Using WXR is fine for any blog that can be turned from XML into PHP objects ... inside the execution-time limit.
Otherwise Wordpress thinks it's just hilarious to silently fail to import anything that doesn't make the execution-time cutoff.
And I don't think I am a big enough fish for wordpress.com to bother to help me do an SQL migration.
These days I've put most of the sites behind Cloudflare, which has the effect that a lot of my media is being mirrored in their Sydney DC, which is nice because most of the readership is Australian -- the same reason I went to the Tokyo centre this time.
From Perth, iiNet routes directly to Asia, and it's a big win. But Amnet routes AU -> US -> Asia and it's a big loss.
It's a big enough issue Australia-wide that Blizzard sells a dual-region Starcraft 2 in Australia, because (ping to battle.net Singapore for some) > (ping to battle.net US) > (ping to battle.net Singapore for others) to a significant degree.
So, being in Perth, I've picked the DC that serves me best. Hmm. I guess I will ask my bloggers how noticeable it is.
I tossed up hosting in Australia, but the bandwidth is just so utterly ruinous.
Somewhere in the middle, and probably closer to the low end, exists a need for less commoditized services to support more complex installations.
It doesn't appear like WP Engine is really positioned or resourced to actually support these needs and this is not the first time I've heard of these troubles.
Look at the pricing: Wordpress VIP starts at $500; WPEngine is selling a $250 plan.
We recently chose not to go with WpEngine because of the confusing help solutions we could see, a few reviews we saw. We ultimately went with ServInt and have been really happy with the move and support.
And to your point the pricing didn't feel right, it didn't make sense compared to similar options (or what we thought was similar)-- maybe I am off base though, it just what we discussed at the time.
This was posted on Reddit and killed with one comment the tl;dr of it : The client was incompetent and irresponsible and likes to blame people when things go wrong.
As nice a guy as this blogger probably is, he's the bane of services like WPEngine. There is no way to service the needs of his complex edge case of a system for anything like $250/month. They are guaranteed to lose money on him, and the only hope they have is to convince him to leave the service as rapidly as possible to avoid being sucked into the time sink that dealing with him is going to become.
This is the reason you have a "refund" button on the customer page of your admin console. A quick email apologizing for not being able to meet the needs of his unique setup and a refund of his payment, as soon as it became apparent just how impossible things were about to become would have solved this completely.
Or at least one hopes so. The tough thing about dealing with toxic customers is that they grow on you slowly and you don't notice at first. Then one day you realize you've spent four hours just digging into issues and writing emails to this one person who you're probably going to have refund eventually anyway.
I've never met the guys on the WPEngine team. But I feel for them after reading this.
Then just tell him that the service isn't for him at that price, like Page.ly did.
If they can't support customers like him, they shouldn't charge him $500 for the time it takes to determine that.
WPEngine are not billing me the $500. WebSiteMovers are.
I approved that work and I will pay that bill, just as soon as they help me out with some technical details. WSM are not in the wrong here.
However I do intend to present an invoice to WPEngine for the amount that I have paid to WSM.
It is part of the mainline code for Wordpress. It is present in the base installation.
WPEngine specifically advertise that they support it.
Also ... since when did we start considering people who expect to get what they were promised by the sales copy to be "toxic customers"?
I don't know what your experience with WPEngine is, but suffice it to say, this post is not the first complaint I've heard, and I've personally had...issues with them as well. And my setup is even more vanilla than the one described in the linked post (which is pretty vanilla, honestly). I love Jason Cohen's contribution to the startup community and I want to love WPEngine, but I am starting to wonder if something is amiss on the technical / customer service side there.
Where did you find this 'complex edge case of a system' in his post? This 'unique setup'?
Why do you think 'impossible things were about to become' when he was able to easily move the sites to WP Engine... after paying a 3rd party to do it?
He didn't describe a single special requirement that I can see. Just plain old WordPress with some existing data to move. Not even a large network or a large amount of traffic. This is supposed to be WP Engine's bread-and-butter customer.
All of WP Engine's plans other than "Personal" say "Hey WordPress MultiSite customer, come here! This plan is for you! Hundreds of thousands of visits a month and 20GB+ of storage!" http://wpengine.com/pricing/
Why are you calling him a 'toxic customer'? He's been running this WordPress install himself on an unmanaged VPS. He wasn't wasting their support staff's time on unnecessary questions -- they were wasting his! That's not a 'toxic customer', that's a 'toxic business'.
I'm having a hard time connecting this comment to the HN submission at all.
If you're looking to upgrade in the future, you might be much better off throwing a dedicated server at the problem and setting a nightly backup to S3. Also, thank you for posting.
Monthly cost is about $40 ... unless you count my time.
The weeks of shitty service beforehand are what really did it.