NOTE NOTE: I used Google to try and get terabyte/petabyte values for all the numbers above. I was too tired to realize it wasn't dividing by 1024 but by 1000. Woops... :)
How the info is processed depends on budget. The NSA had an estimated $10bn+ budget four years ago; I can only assume it's skyrocketed. That's just public estimate guesstimating off of visible operations; the budget itself is classified and I wouldn't be surprised if the (official, ledgered) budget was even higher. So, for the NSA (and similar black-budget groups) you likely have multiple exabytes (tens - say, even hundreds - of exabytes?) of diskspace to work with.
Going off what I assume is saying 16.8Tbps on "Trans-Pacific" at the bottom right of https://www.telegeography.com/assets/website/images/maps/sub..., that generates ((16.8/8)(1024^4))(606024)=177.18PB of data in 24 hours assuming continuous peak throughput, or at least that's what it would generate if that cable is still only doing 16.8Tbps. Given that small(ish) companies like Backblaze can handle 200PB in a fairly small datacenter, capturing all data that goes through that cable for a few hours doesn't sound too unreasonable for the NSA or maybe a few others.
So there's that.
It gets crazy quickly though:
2 days: 354.37PB
3 days: 531.56PB
4 days: 708.75PB
5 days: 885.93PB
6 days: 1.03EB
7 days: 1.21EB
8 days: 1.38EB
9 days: 1.55EB
That's not taking compression and dedupe into account (which will definitely afford huge savings), but it's also not accounting for processing overhead, which will want to generate intermediate datasets (which themselves will be huge) and so forth.
So the main difficulty is time; unless all (presumed) 10 billion dollars is being spent on buying disks, power to run those disks, and invisible ink to make the physical space and power usage go away (however would be appropriate for the situation...? I have absolutely no idea), well, your storage capacity is going to be finite.
So then you need an incredible processing system that, within the storage window you have available, processes the information such that you extract the useful tidbits out and can discard the rest. That's the "the NSA is hiring kids out of grad school!" bit. (And I'm all the way over in Australia and I'm aware this happens. I think I learned about it on here.)
So I'm using the NSA as the "ideal best case scenario" because they have way too much money, are probably doing a bunch of what I'm describing, and this is a fun mental infosec challenge. Obviously your random VPS or VPN provider isn't going to have any this... but oh hey, the NSA is probably tapping those lines anyway, because duh. Remember, of course, that packet switching networks can route data anywhere they want, so it's a toss-up between whether the packets can be coaxed and nudged into going through tapped routes, or whether there are taps on all the routes being used anyway. Same thing or not depending on how you look at it (and the situation).
In some cases the VPNs probably only log to help prove themselves innocent when required, and likely just drop the data without processing it. Some probably do basic keyword or similar realtime analysis; perhaps the ones that have been bitten a lot (I have no idea if there are any, common sense would suggest they would exist).
I would be genuinely surprised if any VPN or VPS providers actually did slightly-behind-realtime complex/involved/high-CPU data inspection/validation. There's simply no value-add in doing that behind the scenes except for more legal paperwork when things are found, and "we perform deep packet inspection on all your incoming and outgoing traffic!" on a homepage is a definite path to zero customers!
Obviously all of this is conjecture and hearsay; I thought of much of it as I went along figuring and typing this out, and I've probably forgotten some avenues of thinking that could result in different conclusions.
I do still remain reasonably paranoid considering the XKeyScore and PRISM stuff. Also, don't forget, at the scale I've been talking about here, I would not at all be surprised if all the private keys for all the CAs your browser trusts are floating around out there (even if you don't understand this, please pass it on). This was interesting to watch: http://video.fosdem.org/2014/Janson/Sunday/NSA_operation_ORC...