Sure, this processing does not need to happen in real time, but it does need to happen.
My point was that to assume the user-visible portion of the site accounts for most of their computing needs is probably incorrect. Video encoding was just one of the first things I thought of, and I bet there are many more.
I admit "processing going on in the background" was a real hand-wavey way to phrase it, but I think you know what I mean. It's an iceberg - there's a lot happening under the surface. :-)