Hacker News Reader

Comment by peterwwillis | Hacker News Reader

I think their blog being hugged to death is a good example of how this technology (read: web backend tech in general, not containers) is still too complicated for people in general to use it for simple cases, like scaling the load of otherwise static content. (It's wordpress, but still) We need solutions that require less resources and smaller learning curves.

View on HN

> being hugged to death

Whenever I read about this I can't help to think that people don't make a good use of a load balancer like HAProxy in front of their web-servers. Even if it's a single server behind the load balancer.

There's this sweet spot where your server is handling requests as fast as it can but if you cross above it everything halts. To make it simple, this usually happens when one or more hardware resources (CPU, RAM, disk) reaches 90% utilization.

So just do some basic benchmarks and find that sweet spot and put a limit in HAProxy. Say 100 concurrent connections for static assets and 50 concurrent connections for everything else.

All requests above the limit will go in a HAProxy queue but they should not stay there for long because the web-server is able to work at full speed without being overloaded.

PS: I have quite a dream. To write some fabulous blog post that reaches the HN front page and serve it from a raspberry pi device. Too bad I don't write at all.

This is informative, thank you. Any HAProxy-like software you recommend? Or is HAProxy the king?

You might be able to climb the ladder with a meta-post. Just do a write-up on how to host a front page post on a Raspberry Pi and tune the headline for the right audience. "This Site Can Handle 1,000,000 Connections on a Raspberry Pi" If you could collect data from the experiment and do a postmortem, that'd be great, too.

> postmortem

Hehe, why so negative?

I've actually dealt with this precise problem, HAProxy doesn't really solve the problem of an excess of traffic. At some point you'll start dropping connections or people will give up waiting.

What you really want is something more like Varnish, a pass through caching proxy that handles known requests from memory, and any novel requests go through.

I agree that Varnish might be a good addition but in the end it depends on the exact setup you're running and which hardware resource is used the most.

My opinion is that the CPU is usually the first to fall and you will not help it quite that much with an in-memory cache when the actual problem is the https overhead or the PHP/DB processing.

What you're describing is queueing, but the solution you're really looking for is back pressure. Any queue will eventually fill and when you fill it, things fall out. So rather than just dropping requests or letting the requests swamp your box, what you want to do is intelligently reject requests.

On the web, this means sending 503 responses. In a more intelligent client-server model, this might be the client doing an exponential back-off retry with jitter. A plain-old retry can swamp the server and just kill everything. An exponential back-off retry will allow the backend to achieve a level of service that isn't fast, but also isn't unavailable.

Hopefully in this case the haproxy is used intelligently and simply returns cached responses to GET requests without query strings. You don't want to hold onto connections or pass them off to the dynamic layer behind the proxy if you're overloaded. You can also respond to the client with a page with an auto-refresh, implementing an exponential back-off retry.

The setup I described will not get you from 10 requests/second to 1000. It will let you to make the most of your hardware though.

I don't know how their blog is hosted but wordpress in general is very chatty with the database backend, and I really don't think the site getting crushed by HN is indicative of fundamental issues with container technology. If it is then there is a long list of technologies that HN has exposed fundamental issues with by crushing their content sites :).

Oh I'm not pointing at container technology. I'm talking web backend technology in general. I mean this is 2018. No well known company's blog on AWS should ever go down at all, unless the AWS service in the region is down. The fact that it's down is not a Docker failure or a Wordpress failure or a Database failure, as there are widely known methods to scale all these things. The problem is that putting the puzzle together is just complicated enough that it's easy to get wrong. If even Docker's blog can't scale, we have bigger issues as an industry.

Fair enough. I took "this technology" to be a reference to Docker given the context.

I'm sorry, is your argument based on the current year? I presume you mean otherwise, but that's how it's written.

Yes, and it's a perfectly fair argument.

Content hosting sites are supposed to host content. That's what they do. If a site doesn't work under unusual but not outrageous conditions, it's fundamentally broken and not fit for purpose.

The industry should be much less tolerant of excuses for tools and systems that perform poorly under load. It's had a couple of decades to get this right now. How is it acceptable that this is even a problem?

Static site generators are an infinitely better choice for making actual blogs. Wordpress has evolved into a CMS similar to Drupal or Joomla with all the available plugins. You can make it into a store or a wiki or basically anything, but you probably shouldn’t. Wordpress was really cool back in the PHP5 days, but Moveable Type and other “old school” static CMS generators always scaled better. Today with node and the dozens of good SSGs, there’s really no excuse to use WP for a blog. The content doesn’t change between page loads, so why would you server-side generate the page dynamically on each page load? The way that it works is fundamentally wrong, and while it’s not very hard to fix with Varnish or WPSuperCache...why not start with the right tool for the job?

Try getting 5+ marketing people to learn git and GitHub sometime and let me know how it goes.

For just the blogging portion, you can just add the blog posts directly through github (the markdown files for posts).

https://help.github.com/articles/creating-new-files/

But yeah, it’s less user friendly. I’m sure someone will make a good solution for this soon.

There are some web services that will run a static site system for you like https://www.netlifycms.org/ or https://www.datocms.com/ .

Gatsby + Netlify + Netlify CMS is super.

Git is not for deployment. Also, hardcoding blog posts in the repo is a dumb solution, and not necessary for a static site. The user built markup should be separate from the code that generates static pages from it.

This has also been my experience when pleading with clients about not using Wordpress... even when they pay me to research SSGs, they seem to always revert back to WP because that's what the non-engineers know how to use.

There are still plenty of CMS options that are built around caching in a way that you effectively get the best of both worlds. I'm a big fan of Grav, which is a flat-file CMS with an optional admin panel.

The aforementioned Movable Type has a pretty graphical interface just like WP. It's just that in the background it generates static files.

It's depressing that 17 years later it still seems to lack a decent competitor.

It just works.

99% of the time for 99% of the people who want it to do something (something realistic), it does it. In order to get people to switch to something like a static site backend it has to be much better than wordpress. It can't be as good, or a little bit better it's got to be waaaay better or it's not worth the time and effort to switch away from something you're familiar with.

If Docker's blog was their core product, I would probably be more likely to agree. The truth is, they don't get ROI from engineering their blog to anticipate or quickly respond to all the traffic of a major announcement, because those are few and far between. The level of complexity is moot because the benefits will only outweigh a very small cost.

They build container tech. That doesn't mean they will benefit from engineering the best solution for everything they do.

I'm not arguing with that, I'm saying it's dumb that we still have to work hard to get a blog to scale.

There is a slight irony in them being an infrastructure company.

WordPress works very well with high loads if you use proper caching (via WP Super Cache, for instance) so the database doesn’t get hit for every page hit.

You don’t need containers or in fact anything fancy for that.

With WordPress in particular containers make some maintenance and scalability issues more difficult to deal with.

I've been mumbling under my breath about this since the 90s. The basic issue as I see it is that nobody wants to do the up front work of separating out static, client side dynamic, and server side dynamic. If that work were done then nominal load would be much less to begin with, and scaling up would be much more straightforward.

Maybe I'm misunderstanding your categories, but why would a server care about the distinction between "static" and "client side dynamic"?

A server would not care. It's the devs who need to separate these things.

Mumbling since 90s? Well, if you stopped mumbling and showed the world how it ought to be done, perhaps we would be doing it your way. Show us how.

We need more static page generation like Jekyll. It doesn't make sense that an entire blog needs to be dynamically generated all the time. Dynamic contents can be fetched using ajax, but the rest should be static HTML.

Static content can and should be cached, not necessarily pre generated. Thus you can use your site as is and handle caching with whatever you need. It also takes care of semi static (slowly moving).

I don't understand why companies take on hosting their blog and marketing page (usually the homepage) when it isn't their core competency and a distraction for the engineering team to maintain.

It looks like this is true for Docker which seems to be hosting it themselves instead of using Wordpress.com

» nslookup blog.docker.com Server: 192.168.86.1 Address: 192.168.86.1#53

Non-authoritative answer: blog.docker.com canonical name = blog-1556129762.us-east-1.elb.amazonaws.com.

Or use WpEngine or FlyWheel (https://getflywheel.com).

well if a company which tries to sell solutions for high scale , high performance solutions and fast rollout and scaling. hosts their own blog (instead of using a hosted solution) AND can't scale it to their needs, than I would say their product failed. Either because they couldn't use it in this case (because too hard, etc, whatever) OR because it basically didn't worked.

They are a container company, not a scale your blog + scale your database + scale your cache + scale your security company.

Than they should‘ve hosted it somwhere else instead of on their own tech

See my first comment.

When did that become so hard -_-

Because a blog at scale is still a complicated piece of software? You need to implement caching, DoS protection, security - something a hosted blog vendor might have experience in.

What do you mean still? It's about the easiest thing you can do... It was easy a decade ago and I reckon it's easier now. For Wordpress it can be literally a few clicks. I'm assuming they're not serving millions of users, that's when "scale" kicks in.

Or it's completely unrelated.

What does this have to do with containers? They just need a CDN in front of the site.

I don't think Docker is too complicated; instead, I think a lot of companies misunderstand the benefit of containers aside from "my application boots faster"

Maybe he originally set up the blog using his work login and HR auto-deleted his credentials or account. Sounds like something that would happen around here...

Reply on news.ycombinator.com