BBC Digital Media Distribution: How we improved throughput by 4x | Hacker News Reader

BBC Digital Media Distribution: How we improved throughput by 4x

bbc.co.uk

79 pointswoolie23 comments

Varnish author here.

I'm several sets of information short of being able to say anything particularly precise about their results.

Their diagnosis is very detailed and useful, but I think the main thing we can learn from it is that the Linux kernels VM system is horrible when memory is overcommitted. At least to me that's not news.

I will also say that we in the Varnish project have not worked much on performance the last couple of years, because people have asked much more for features than performance.

Some of the stuff we have done have traded CPU cycles for flexibility, but if the Varnish users change their mind and decide performance needs a boost, we can do that too.

The reason we dropped sendfile(2) in Varnish was that nobody seemed to be able to show any credible performance improvement, and dropping it allowed us to implement on-the-fly processing of the HTTP bodies in a cleaner way.

We can certainly bring sendfile(2) back for "the straight" path if that's important to people.

The "file" storage was our first rather primitive storage engine and it has never been optimized to any extent. Varnish Software Inc. has a commercial offering that includes a "massive storage engine", but most people seem to just use the 'malloc' storage.

And no, Varnish is not dying just because BBC doesn't use it in the absolute far corner of high-end usage doesn't use it: If NGINX is the better tool for them, they should absolutely be running NGNIX.

Direct link without two tracking redirects: http://www.bbc.co.uk/blogs/internet/entries/17d22fb8-cea2-49...

This is really interesting! But keep in mind, Varnish is blazing fast if you manage to fit everything in memory by using malloc. Also Varnish is more configurable than the Nginx cache. For example if you need to serve different content on the same url based on the Accept-Language header or a cookie, you can do this easily with Varnish by creating a lookup hash of the url + header value in VCL.

Came here to say this, feels as though they maybe didn't do Varnish justice in some ways.

Kudos on the huge improvement though, it's the results that matter not the means.

Seems to be that they did, but Varnish didn't suit their needs as they couldn't fit all the data in RAM. It's OK to change what you use because you outgrow it.

You can do the same on Nginx with https://github.com/openresty/lua-nginx-module (and much more)

Absolutely, Varnish with malloc has (and still) served us well when the working set fits in RAM - that was why we started with Varnish. We still use it in front of our web servers and will for the foreseeable future!

Not that it's a bad article (lots of nice detail in there), but the short version of this is: if you're web-serving disk-scale things, use a web server that's designed to serve from disk. If you're web-serving memory-scale things, you can use a memory-based web server. Even if you think you're just "caching".

Interesting bookend to this [1], which felt like the best possible endorsement for Varnish I'd seen, and was a driving factor in me building it out where I work. As others have pointed out, it seems more like an issue of being the wrong tool for the current needs, and not so much a problem with Varnish itself.

1. http://www.slideshare.net/grahamlyons/varnish-at-the-bbc

Its worth pointing out that the BBC continues to use Varnish in the way Graham described in that presentation.

As you say, its about finding the right tool for the job.

Actual URL: http://www.bbc.co.uk/blogs/internet/entries/17d22fb8-cea2-49...

Interesting line of reasoning. Definitely mmaping 8.6TiB into user space process virtual memory is gonna take a toll.

I tried to find why varnish doesn't use sendfile: https://www.varnish-cache.org/lists/pipermail/varnish-dev/20...

  By default [Varnish uses sendfile] never; there are problems with all sendfile() implementations

In the past I've found that sendfile is not portable. FreeBSD and Mac OS X require the file that is being sent be a regular file[0] while Linux will take any file descriptor[1]. That being said, I can't imagine what benefit memmapping would have in the context over the file system page cache.

0 - https://www.freebsd.org/cgi/man.cgi?query=sendfile&sektion=2

1 - http://linux.die.net/man/2/sendfile

I'm surprised about the 6-year-old kernel on a server with those specs. Apparently its still maintained until early next year. Anyone know if that might play a factor on the benchmarks?

EDIT: nvm they did try other kernels but it only made minor changes in results.

> The RAID 50 array appears to be the current bottleneck.

Any form of parity RAID is always going to be a bottleneck, and should never be deployed into any production environments this side of 2010.

If your application can gracefully handle IO balancing (and potentially replication) between individual block devices, and handle them dying in all of the weird and wonderful ways that they do, then go with JBOD. If not, then your choices are a mirror (E.G RAID1), or striped mirrors (E.G RAID10).

Yep, we went JBOD and saw the performance win as expected :)

> dying in all of the weird and wonderful ways that they do

It's only been a few months, but there have been plenty of great surprises thrown out by those disks. Both in the ways that they've failed and how the kernel has handled those failures!

That sounds like having wrong organisation of data and then using wrong tools to do the job. Interesting article though.

What I want to hear from BBC is not how fast they can make a web server, but what are they going to do about those 1 million hours of archive TV and radio content they are sitting on. It's a shame for that treasure to be locked in copyright limbo, while YouTube collects 100 hours of cat videos per hour.

> What I want to hear from BBC is not how fast they can make a web server, but what are they going to do about those 1 million hours of archive TV and radio content they are sitting on.

So nobody at the BBC should be allowed to say anything publicly unless it's on that topic? That's idiotic. The guy writing this blog post probably has absolutely no control over archive content at all. Why should he be silenced?

I used to record Jools Holland onto VCR every week, and then make mix tapes of my favourites. I would happily pay 99p a track for my favourites - and feel like a winner.

Simlarly with live lounge sessions. We get a selection of what some pratt at the BBC thinks were the best of the year in a compilation CD. WHY NOT JUST SELL THEM ALL?

I have been bitching about this for years, how has it not been made a thing?

I think the problem with music broadcasts (TV or radio) is that they'd need permission of the artists and/or publishers before they could sell it commercially. It's probably for similar reasons that they don't show all the Glastonbury stuff (mixed in with revenue sharing - some artists/publishers might want too much, for example):

> "On occasion an act may not wish to be filmed or recorded. Artists may also agree to be recorded but only allow a limited number of songs to be aired. This could be for a number of reasons for instance: the quality of some parts of the performance, because they do not wish to broadcast new or unreleased material, or they do not want to broadcast their entire live set."

I do agree that it'd be a good idea to do, though.

I mean, obviously artists could opt out (and tracking them all down for back catalog perms would be a nightmare - I'm not without sympathy for the effort involved) but going forward could they really not set up a default agreement? The fact that they DO release some stuff commercially seems indicative it's entirely possible.

Glasto is indeed another great example of audio tracks I'd gladly pay for - funnily enough though by the time the BBC takes it's coverage down some valiant internet person has stuck what I want up on torrents.

Blame those that own the copyright (i.e. the labels). The list of what tracks can be used will change weekly, if not more often, and offering them for sale will be an even bigger headache than being able to play them on the TV/radio.

Reply on news.ycombinator.com