Observability is not only for SREs
lightstep.com
lightstep.com
If you expect people to use a tool for your benefit, you’re going to be disappointed and/or bamboozled every single time.
Managers are experts in this. They don’t seem to see why they are always having to harp on people to keep visibility tools up to date because “not getting harped on” is not a motivation to do something. Most of my job is keeping track of my responsibilities. If I’m keeping track of yours, then why are you even here?
People use tools for their own benefit (if even then). If that helps out someone else, that’s a bonus feature, not a core one. So we use testing frameworks if it helps us stop our own regressions. We use issue trackers to fight pre-emption and weekend amnesia. If you don’t honor that (by mandating that they be used in a different way that raises the cost/benefit ratio) then you’ll get less compliance, not more.
So if the SREs want something, they better have a self-interest angle, otherwise they’ve just got a platform for lecturing, not for progress. Of course, there are a few people in any org who are satisfied with lecturing.
Re: SREs and observability, unfortunately sometimes the power imbalance tips the scale in their favor because if as a service owner you let observability out of your control then guess who will report your SLA to your boss?
Of course every org and every SRE org is different. Except yours. The Anna Karenina principle applies.
Ive had managers who have understood this dynamic and others who have ignored and try to use force by continuing to harp after Ive provided reasons. Generally I haven’t lasted that long under the latter.
I installed a graphite/grafana cluster[1], and delivered a bunch of tech talks on how to massage raw metrics into useable metrics that related to business goals.
I did this because I hated waiting for splunk graphs to generate, and I could never really get any kind of actionable graphs out of it.
over a couple of years it went from a couple of thousand metrics to well over a million active metrics. (anything not updated in the last 7 days was deleted)
but the biggest "oh its actually useful" was seeing product owners, scrum leaders and buisness analysts create, update use dashboards, and setting alerts on service health _without_ software engineering help. It was brilliant, and a testament to how simple graphite/grafana is to use.
[1]well, I installed a graphite cluster, someone twisted my arm to get grafana installed.
Don't metrics have to be defined first? I assumed you'd first do a big data log analysis to understand what is going on and then monitor for metrics learned to be useful. Storing just metrics means it'll be hard to investigate when things go wrong when the currently defined metrics don't capture that new issue.
In big data circles the consensus these days is to store all raw data since storage is cheap. Useful cleaned aggregations and metrics can be extracted later by processing the full history using the now available big data tools.
I suspect this approach is a problem in the traditional software engineering, where focus is not on historical data like it is in data engineering. The solution goes - Splunk is too expensive, let's ditch logs altogether and hope we capture some metrics and plot them in nice dashboards. Instead of dumping the logs highly compressed into cheap s3 and running some Snowflake or Spark on it later. Heck, even dumping the intra-day non historical logs into Kafka or Materialize and extract the same metrics on the fly while preserving all raw data.
With that being said, it's not a mutually exclusive situation. You can have both. However, some logs used for plotting metrics have near-zero debugging value (e.g. a log line that just includes the timestamp and the message "event x occurred"). Those kinds of logs should be fully converted over to metrics.
Some other logs however are genuinely useful (e.g. an exception occurred, the error count should be incremented, and this is the stack trace).
What is the difference between logs and metrics then?
Logs are pieces of text and metrics are well defined data points?
I can represent both using the same log format. I can also extract the logs with well defined structure into a columns store or inverted index for fast querying and plotting.
The entire distinction of logs and metrics and keeping one vs the other reeks of strong premature optimization by the software community. Storage is cheap, just dump the raw logs to s3 and run etl on them to extract meaningful metrics.
Logs, metrics and traces have the same representation - text or some kind of json yo have it structured. Metrics are just logs with well defined shema. Traces are logs with correlation ids in them to allow for joining between logs coming from different services.
It's just a data problem, nothing else. However, people keep overpaying for complicated services for observability that they don't need. When they split it into logs, metrics and traces they have to connect their programs using proprietary connectors to these external services, adding vendor lockin and potential for failure when the observability service has downtime. Instead of just dumping logs into stderr as json objects as intended by the unix philosophy.
Likewise, traces are indeed (as you point out) functionally just correlating logs from multiple services which is akin to syntactic sugar. But again, that's precisely its value _at scale_: easing the burden of use. I've personally seen traces that reach across tens of services in an interconnected and cyclical graph of dependencies that would be hellish to query the logs for by hand.
From my second FTE position I’ve been dealing with the consequences of people logging things that were important to them a year ago that they no longer care about but can’t be bothered to remove either due to hoarding dynamics, project scheduling dynamics, or effort/reward dynamics (“that’s an old feature/bug, and I get paid to work on new things”).
Stats get aggregated with their own kind, while logs get jumbled together with everything else. The median value of old log messages is negative, while that of old stats is closer to zero.
If you put a bunch of data into a log message, its initial value is higher but so is the negative slope of the line. If you don’t put a bunch of data into the log message, what’s the difference between logging it or recording a stat?
Part of the unspoken assumption of logs is that I can read cause and effect chronologically in the logs. That ceases being true the moment you have a distributed systems. Getting reliable sub-second clock skew is expensive. Getting it down below speed of light delays within a single rack is pure fiction.
Anyone implementing stats on a single server application has bigger problems than logs vs stats, so distributed is practically a pre-req for even having a stats system.
I find it much harder to guess what log messages I will need in the future compared to what metrics.
Metrics are easy: for any resource, service, component, and queue (explicit or implicit), you want to know cumulative arrivals, departures, timeouts, errors.
That takes you most of the way there for most things.
You can add counters for significant events too, but they tend to be automatically included by the above. (Basically departures from the component that trigger the event.)
I don't usually promote on HN but this is exactly why we built https://axiom.co! We've been working on this problem for some time, essentially allowing schema-less/index-free ingest, S3-based storage in a highly-efficient format, and then querying with a Splunk-like (specifically Kusto-inspired) language via serverless functions.
We built it because we also realised we would avoid or think too much about logging (cost, scaling, retention, etc) which led to compromises either in our monitoring or later when we wanted to dive in and try and draw some insights/analytics from that kind of data.
This is a _very_ good question.
Getting from log to meaningful dashboard full of metrics is very hard to do well. Its also very hard for a normal person(ie non programmer) to ask the right question of a log stream.
For example if you give me a log stream of a web server, and want me to raise an alert when an instance's health check takes longer than n milliseconds, it would require regex to get the host name, regex to get the health check, and more regex to get the response time. All of those steps are hard and require testing. Moreover its fragile, any kind of format change is really easy to slip in and cause your monitoring to break.
So your next question might be, why replace splunk with a metric based solution?
Thats the thing, you don't! logs are really really useful for what specifically went wrong. But they are not very good at telling you what is _currently_ going on over n number of services.
when implementing a service/program/things, I ask the developers to think about the information that is useful to log. Be that response time, number of peers connected, cache size, etc etc. Then instead of just dumping them to logs, you'd push them to a metric library and get that to look after it.
this means that you are consciously thinking about an number that best sums up the thing you care about. This then frees up logging to explain _why_ something has happened, rather than being rigidly defined because you'll break a dashboard.
Think of is as this:
dashboard: shows you if a service is happy, which is an aggregate of many smaller programmes.
Something goes wrong, response time of a microservice is above limits, you then track down when exactly the metric went wrong, and fire up splunk to look at the logs.
Kudos on you for introducing them to a metrics pipeline though!
Wait...there are companies where there's still a world of QA?
https://en.wikipedia.org/wiki/Mirror_Universe
But you can read "QA" as "what you'd do to investigate the system if you had time and didn't have people constantly pushing new feature tickets at you."
A good QA department will know what the software does, but more importantly how it should behave, and how its changed over time. They should be a good resource for working out how a feature should look and behave.
a good QA department is worth its weight in gold
On the other hand, we can look at SW testing as "building a robotic user that will catch errors before real users". This approach is far more intellectually-engaging, productive and gets SWEs in touch with the product.
That is to say, SWEs aren't shit at "testing", they are shit at product, because product is virtually omitted in traditional CS curricula.
(This is the more modern version of QA, of course, where it stands for Quality Advocate or Quality Assistance)
Now: they write code, they test it, they understand customer’s requirements, they monitor their code, they are on-call, they mentor others, they challenge the status quo, they give tech talks, sometimes they attend company hackatons…
I’m tired of software engineers having to do a bit of everything.
Not true. There's nothing magical about testers. They don't have "special" skills or deep voodoo powers. The best ones are just smart people who understand people and software.
> They should be a good resource for working out how a feature should look and behave
You must have weak designers and PMs. Everyone should be welcome in the process, but those people drive that stuff.
The thing to remember is that getting rid of QA upsets a triumvirate power dynamic. If it’s the dev team versus project management, the dev team always loses. If it’s sometimes dev and QA against PM, then you can win those fights. But only if you haven’t removed the QA manager from the org chart.
We’ve played ourselves in essentially a union-busting manner.
If you’ve always been at loggerheads with the QA team, I can tell you that you’ve been missing out. Looking back, defusing the default animosity between QA and Dev has gotten me more promotions than anything else in my career, and the lack of a QA team has put a substantial damper on my upward mobility. It sucks and I really wish we could go back.
Some do, seriously.
I'm coloured by video games, but some of the QA people I've seen are magic, without them entire products wouldn't ship at all.
There's something to be said for having "break shit and report on how you did it" as a job title, regardless of the other gains you get for having someone who holistically understands the product.
Some are even lazy, so work to automate themselves out of working, which makes their impact much wider, even when it's domain specific.
It's also the case that the people coming out of QA in gamedev and going into other parts of the company tend to do extremely well compared to people going direct into that area. Our old managing director started in QA, as did the executive producer.
There is a spectrum, of course. But, a good QA has a "feeling" for finding where things break, an excellent QA know how to fix it.
Just as writing software is a skill, so is QA'ing along with Sysadmining.
Yes, there is: they get to do testing full-time. Engineers spend their best “brain time” designing and writing systems. Testers spend their best brain time, well, testing. That can make a huge difference when it comes to find bugs.
Oh, but there is: they have fault-finding in their job description.