We compress Pub/Sub messages and more, saving a load of money
blog.lawrencejones.dev
blog.lawrencejones.dev
If you're interested, there was a nice discussion in /r/devops about this the other day: https://www.reddit.com/r/devops/comments/kmltbx/how_we_compr...
I don't think it's yet deployed in browsers, so I'm restricting my recommendation to tools and backends. IIRC brotli is worth considering for browsers, but I haven't deployed it myself.
60 TB is a lot, but we can all agree that is surely doesn't warrant -/+ 13000 USD one-way trip price tag.
This statement is in particular worrisome: "Google Compute Engine network rates (we’ll ignore these, as they get complicated)"
So you are basically buying a black-box that has an ability to drive your business into a ground if you are not careful enough.
Yes, I am aware you can argue and plead for mercy once you get the bill, but this is not the way: they wouldn't be so keen in forgetting your cost if it was making them a real expenditure: I was doing a napkin calculation for AWS and I found it to be around 10x more expensive than DIY solution, which means that from 10 suckers, only one has to pay for them to make profit.
The more I read about cloud providers the more I am convinced that their major business is based on the fact that they have to get hardware and software for their core business and since they cannot scale it precisely it is more lucrative for them to just try to sell "solution" to suckers for 10x and get rich in the process.
[1]: https://www.2ndquadrant.com/en/blog/on-the-impact-of-full-pa...
It's pretty low volume actually. A small company with < 100 developers and servers can generate a billion logs over a few weeks.
Normal logs from the system, syslog, applications, databases, web servers... nothing fancy really. It's common practice to centralize all these into ElasticSearch or Splunk.
Their scale of 10 billion logs 60 TB means they're a regular small to medium company.
I work in a tightly regulated industry (finance), and even my company doesn't have a need to log 25 messages per second per person.
Is anyone else able to validate this claim that regular small companies log this much data?
This logging system was for all https://gocardless.com/ systems. We're a B2C company which means we have different economies of scale than many scale-ups of our size, but you were close with your guess:
Currently 450 people worldwide, ~150 in product development, of which ~100 fulltime developers.
It includes:
- System logs, such as our Kubernetes VM host logs, or our Chef Postgres machines
- Application logs from Kubernetes pods
- HTTP and RPC logs
- Audit logs from Stackdriver (we use GCP for all our infrastructure)
> do you ever do anything with them that requires you to store so much data rather than just a representative subset?
Some of the logs are already sampled, such as VPC flow logs, but the majority aim for 100% capture.
Especially for application logs, which are used for audit and many other purposes, developers expect all of their logs to stick around for 60d.
Why we do this is quite simple: for the amount of value we get from storing this data, in terms of introspection, observability and in some cases differentiated product capabilities like fraud detection, the cost of running this cluster is quite a bargain.
I suspect we'll soon cross a threshold where keeping everything will cost us more than it's worth, but I'm confident we can significantly reduce our costs with a simple tagging system, where developers mark logs as requiring shorter retention windows.
Hopefully that gives you a good answer! In case you're interested, my previous post mentioned how keeping our HTTP logs around in a queryable form was really useful for helping make a product decision:
If you mean marking messages as having been compressed, then absolutely yet. The Pub/Sub messages were tagged with `compress=true|false` so we could update downstream and upstream independently.
If you mean buffering log messages into a batched 'frame', then yes we did do this. We were taking about 500 log entries and compressing them into a single message, which was part of why the compression was so effective.
If you mean something different, then I'm at a loss and very interested in what you meant by the term!
a quick test copying a 24M file (with similiar compression ratios) to s3 showed a 6% decrease in cpu time when piping through gzip.
Gzip is in the order of 10 MB/s with default settings, down to 1 MB/s with the strongest compression setting. It's really really slow.
slz is closer to level 20 (if there was a level 20). It's fast but the compression ratio is meh. You're better of using lz4 or zstd.
I wouldn't normally expect gzip to be a net savings (it's comparatively more expensive), but depending on compression ratio achieved and what layers you're passing the bytes through, I'd definitely believe it can be in some contexts.
60TB of logs was 60 days worth of retention, so 1TB a day. That means we process about 11MB/s on average, peaking at 100MB/s.
A single CPU core can manage 100MB/s of compression, so if you assume we compress and decompress in multiple places, let's say we're paying about 4 CPU cores constantly for this.
That's a pretty worse case scenario, and it would cost us $37.50/month on GCP for those cores, in order to save about 100x that amount.
The takeaway (for me at least) is that compressing an in-flight transfer is almost always worthwhile if you're working in the cloud and the transfer will eventually be stored somewhere. The economics of CPU vs the total amount of data storage cost is a no brainer.
Hope that makes sense!