Hacker News Reader

Comment by ceejayoz | Hacker News Reader

I shudder to think how much Splunk costs at Twitter's scale.

This is how you get organisations that pull in $3.7 billion in revenue but still can't turn a profit :)

Twitter has been profitable for the last several quarters, FWIW.

I am pretty sure Twitter gets a hufe discount. A company like Twitter is on Splunk platform itself a huge advertisement for more future traditional Fortune 500 companies.

This blog post read a little like an ad itself to be honest.

It probably gave them a 7 figure discount

We log >>100TB/day in splunk and get all the discounts. It’s still ridiculously expensive.

Many of the issues presented in the article ring very true. Splunk is pretty amazing for adhoc analysis/threat hunting. However, once you know what you’re looking for the value proposition drops precipitously.

Can you give a ballpark figure? I can’t even begin to imagine how much that costs.

All in (software/hardware/people) ~$1-2k per terabyte of data indexed.

what kind of use case can log 100tb? Do you actually query this data later?

The whole architecture looks insanely expensive. Putting your debug logs through scribe, through Kafka, then into Splunk, and indexing them, has got to cost something like fifty dollars per gigabyte.

I've always been a proponent over leaving logs where they were produced, not collecting them, and not indexing them, so an architecture like this I find just shocking.

When you have fleets of thousands of machines performing a given service, and thousands of services implementing your product, you simply must centralize and index logs if you want to have any chance of managing outages.

Experience suggests otherwise.

Interesting. What company have you worked for that runs hundreds of thousands of servers without centralized logging?

Not the person you replied to, but basically airlines, banks and other financial institutions, most things government or DOD are largely run this way.

How many servers do airlines and banks run? And how do they get logs?

+1, and how many engineers do they employ? thousands, just like big tech companies.

I worked on a tier 1 service at Amazon with over 1500 hosts in the fleet. Logs within the hour would be on the host and we would literally just grep through them, to grep through logs on a subset of the fleet or the entire fleet there was a simple utility that would ssh into multiple prod host and run grep in parallel. For logs older than an hour the logs would go gzipped into storage and the we zgrep'd though them. No centralized logging (unless you call logs being rotated off the host at end of the hour) and definitely no indexing.

Never had any problems managing outages.

Grep alone can go a long way in managing outages.

All the resources are at the edges of your infrastructure, so distributed grep makes more and more sense the larger your installation becomes. Centralized logging looks less and less economic as things expand.

Thanks, this is a great insight. I haven’t thought about it in this way and it sounds like a very pragmatic approach without over complicating things.

Did you ever worry about those greps interfering with production performance?

Do you ever worry that your logs exporting agent interferes with service performance? After all, that goes on continuously in a setup like the one in the article, rather than as-needed in a distributed predicate evaluation setup.

A constant stable performance drag is going to show up in your load tests, capacity planning, etc. rather than in surprising intermittent degradations.

Also our log collector agents run in containers like everything else, so there is some amount of resource isolation (not perfect of course).

Not really. We only had the option to grep logs on prod hosts only for the current hour so the only time that happens is when there is critical issue going on, logs being limited to 1 hour also means they were limited to a few 100 MBs tops. Even my old 2014 era thinkpad can handle that workload without breaking a sweat. It didn't even cause a blip in our metrics. Grep is a well written tool.

All the log grepping for data older than the current hour happen off-prod host and thus was never a concern.

Interesting. I worked at Amazon on a large CDO product with hundreds or maybe low thousands of services. Some services probably had a couple instances, others had hundreds. There was a customized ELK stack that was indispensable, in my opinion, to tracking problems and communicating them (e.g. here's a URL that anyone on the org can look at). I'm trying to imagine distributed grep working as a solution for that org. Maybe it could work, but the large number of varied fleets owned by dozens of teams makes it a bit of a different problem.

Did your org stand up their own elk stack? AFAIK when I worked there the only centralized company wide log solution that was available was RTLA and RTLA was designed for detecting and alerting Fatals/Error/Exceptions in logs and not a general purpose log analysis tool like Splunk enterprise or the ELK stack.

I could see orgs standing up their own solutions like ELK but our service didn't have to. We just relied on grepping logs stored in Timber for logs older than an hour and grepping logs on prod hosts for real time searching during outages. Granted, our service did not have many dependent services but AFAIK the retail website which has tons of dependencies also followed a similar model (along with using RTLA for fatals) atleast at that time (circa about 3 years ago).

Centralized logging is one of the many problems you sign up for once you opt for microservices on a cluster scheduler. Since service instances are ephemeral and cluster nodes only slightly less so, it doesn’t really work to leave the logs in place.

In enterprises of this size, it’s quite possible you get a request to debug something that happened a month ago. Your instance may be gone, so you need to debug going only by whatever was logged.

Can’t do that in cloud (i know twitter doesnt run in cloud) and autoscale - machine goes poof and so do the logs. One of the reasons why autoscaling to save $ is bs.

That’s simplistic. For one thing, at scale not every debug statement is a precious snowflake. Secondly, in an orchestration scheme like k8s there’s no reason why your container lifecycle can’t include a cleanup container that runs and either exports the logs or just sleeps for an hour, so ephemeral data lives a little longer.

Exports your logs to where? I would call that place centralized logging.

I’m assuming neighboring nodes. Now you’re basically designing replicated (best effort), distributed index - a mini version of google search. This is a clever design but number of companies that can diy it with sufficient quality I can probably count on one hand

For the price of even a modest splunk enterprise setup you can hire half a dozen top notch backend/distributed systems engineers. That’s the opportunity cost of a setup like the one in the article.

I’ve heard this before and it sounds like Splunk’s margins are just way too high. Why doesn’t someone sell a logging platform that’s cheaper than writing your own at scale?

Right if talking top notch that’s 3-4M/y alone. Admittedly I’m completely unaware what the splunk’s ballpark rate is (i heard it’s quite high) but still I doubt that very many companies are at that scale. Also in the current market convincing execs to save on something that doesn’t translate directly to cogs can be a challenging proposition =)

Yes you can try to do bunch of hacks like that, but even just VM failure rate in cloud is insane compared to any modern hardware (looking at you, gcp, but aws too and lower tier providers are even worse). I see multiple VMs drop ded every week with no way to save those logs unless you use PDs ($$$)

I was thinking something along the same lines.

The article said 42 TB per datacenter. How many datacenters is Twitter running?

The article said three datacentres further down, so they're capturing a total of around 126TB/day, which is big but definitely a long way short of the biggest Splunk deploys (multi-PB afaik).

Egress filter before logging to Splunk people, hell if you can practice filtering better from the application itself.

No need to say send all you debug logs to security, or all your info logs to a logging instance for site reliability.

I’m fairly certain it’s more than $1,000,000/year

I'm fairly certain there's at least another zero.

Reply on news.ycombinator.com