Hacker News Reader

Comment by flurly | Hacker News Reader

Relevant posts about how other companies do it

Cloudflare - https://blog.cloudflare.com/http-analytics-for-6m-requests-p...

Facebook - https://research.facebook.com/publications/scuba-diving-into...

For those who are smaller and don't have the money to pay for Splunk enterprise, and don't have the headcount to build your own logging infrastructure, I built a a service called GraphJSON that makes it super easy to log and process any type of data. You can read more about how and why I built it here https://www.graphjson.com/guides/about

View on HN

Thanks for the links!

I was hoping TFA would break down their log-based observability strategy and go into things like trace ids, structured logs....

Instead I am disappointed to learn that engineering at Twitter still sounds.... suboptimal. This from the company that brought us the fail whale (I can't find the blog post now, but I recall Twitter's instability was due to a massive kludge that went unnoticed for years).

Just a nit, for Facebook, Scuba is not the logging ingestion service. It's Scribe, which Scuba itself uses as the ingestion :)

https://engineering.fb.com/2019/10/07/data-infrastructure/sc...

(I worked in Scribe)

It's a pity that scribe is both astonishingly great and totally abandoned.

Another nit, the "abandoned" part is the one that was open sourced a loooong time ago. Since then, it underwent a ton of changes but yeah, it's within Facebook. So, not abandoned at all, but not available outside.

The main issue (as usual in big companies) is the large amount of inter-dependencies with internal systems. Scribe as-is today doesn't make much sense outside Facebook. And yeah, it could be cleaned up, mock the internal services and OSS it. But that's a lot of work, both doing it and maintaining it. And having all those mocks, etc "wouldn't be Scribe as-is either"in terms of how e.g. it scales and so on.

In any case, the storage (LogDevice, https://github.com/facebookarchive/LogDevice) which is a large part of the system was open sourced a while back and... sadly it went again into "not-maintained OSS".

Finally, you can get a similar-ish system working with OSS tools (e.g. fluentd + Kafka) that will also scale quite well and which IMHO can also made to scale to Facebook-size levels. So, the incentive is there, but there are OSS alternatives already available :)

Well, the open source part of it is totally abandoned.

The internal improvements aren't interesting or available to anyone outside of facebook.

The open source version is creaky and very difficult to drag forward to newer versions of thrift.

So -- thanks Facebook for dropping weird artifacts like the aftermath of Roadside Picnic. Woe unto all who adopt these soon-to-be-orphaned technologies.

Just started my 2 week trial of GraphJSON a few days ago! I like it so far, kudos. Unrelated question: do you have some kind of notification setup for HN? I feel like I’ve seen you comment on many logging-related posts, so figured I’d ask =)

Thanks! I do use f5bot, but in this case I was just perusing HN and figured I'd drop some links and a plug :)

Reply on news.ycombinator.com