The real takeaway here: Honeycomb is a great tool for log and trace analysis, and by using Honeycomb's minimal data model [1] for traces you can expand tracing throughout much more of your stack. Now Honeycomb itself makes you pay by the event so Slack's internal buffer and collection system really lets them use it effectively for just the data they need. The low adoption of tracing is also due to the high cost of application level integration, which they are deferring by using the existing tracing systems service layer integration.

So if you come at this as a Honeycomb user Slack has not really re-invented any wheel so much built their own backend to funnel data to Honeycomb and use it's excellent trace viewing and light weight analytics UI [0]. In fact API and model which this article goes to great lengths to explain and justify is basically the Honeycomb tracing API and data model [1]. Which itself is an evolution of the system Facebook uses [2]. It is very cool that they have plugged the gap in Honeycomb which is the lack of historical analysis capability by also funneling all data into Presto.

I would love to see the system they use for keeping field/tag names under control. Honeycomb really strains under large numbers of field names.

0 - https://www.honeycomb.io/trace/

1 - https://docs.honeycomb.io/getting-data-in/tracing/send-trace...

2 - https://www.techrepublic.com/article/ex-facebook-engineers-l...