Monitoring Is a Pain

- mistake #1, assuming it's a technical thingy that some techie person should do and they should just get on with it and do it. Whatever it is. The mistake here is that without guidance this person is going to be selecting some random tools mostly focused on operational things (logging, infrastructure). These have low value for other stakeholders. Yes you need that but these are also commodities.

- mistake #2, each of the stakeholders adds their preferred tools to the mix and the end result is a bunch of poorly integrated tools with a lot of complexity.

- mistake #3, assuming that selecting a tool means the job is done. It's not. Most of these tools assume that you have some notion of how to use the tool and what you want out of the tool. Getting the tool is a the beginning, not the end.

The combination is lethal. Lots of complexity, lots of data, lots of cost, and not a whole lot of value being delivered.

The fix is an observability mindset that exists in product, tech, and business departments. Select 1 tool, not 2, 5, or 10. The more tools, the more fragmented the effort and the more duplication of effort and mistakes. Have a plan, engineer for the system to be observable, have a notion of things that are important to track, etc. Don't dump the responsibility on some techie but actually reflect a bit on what you are looking to get out of this and what this is worth to you.

o Logs are there for ignoring. You need them precisely twice: once when you are developing, and once when the thing's gone to shit, but they are never verbose enough when you need them.

o Using logs to derive metrics is an expensive fools errand pushed by splunk and the cloud equivalents(ie cloudwatch and the like). Its slow, inaccurate and horrendously expensive.

o using logs for monitoring is a fools errand. Its always too slow, and really really fucking brittle.

o metrics are king.

o pull model metrics is an antipattern

o Graphite + grafana is still actually quite good, although time resolution isn't there.

o You need to raid your metrics stores

o We had a bunch of metrics servers in a raid 1, which were then in a raid 0 for performance, all behind loadbalancers and DNS Cnames with a really low TTL.

o Cloudwatch metrics are utterly shite

o Cloudwatch is actually entirely shit.

o tracing is great, and brilliant for performance monitoring.

o Xray from AWS is good, but only really for lambdas.

o tracing is fragile and doesn't really plug and play end to end, unless you have the engineering discipline to enforce "the one true" tracing system everywhere

but what do you monitor?

http://widgetsandshit.com/teddziuba/2011/03/monitoring-theor... this still is canonical.

In short, everything should have a minimum set of graphs, CPU, Memory, connections, upstream service response times hits per second and query time, at a minimum.

You can then aggregate those metrics into a "service health" gauge, where you set a minimum level of service (ie no response time greater than 600ms, and no 5xx/4xx errors or similar) red == the service isn't performing within spec, yellow == its close to being outside spec, green == its inside spec.

if you are running a monolith, then each subsection needs to have a "gauge". for microservice people, every microservice. You can aggregate all those gauges into "business services" to make a dashboard that even CEOs can understand.

In place of argumentation which would be too long, let me give you an analogy. Monitoring is telling you the fire alarm went off. The exact reason is left for a investigation and that's where saving data like logs and metrics helps. But first you need to evacuate the building and call the fire department. If instead of detecting the smoke (e2e) you try to monitor all the data like pressure in gas pipes, all the pipes fittings, how many lighters are in the building and where, etc, you will wake up every night 10 times for nothing and when a fire starts you will burn sound asleep.

Oh, and don't try to scale Prometheus, that's what things like victoriametrics are for.

There are a lot of different solutions, that have different trade-offs for different volumes of data, cardinality of data, how out of order / async the data may arrive, whether you need push or pull delivery, etc, etc.

These apply to you and only you know the parameters and considerations of your application, and the environment it runs in (perhaps you come with the baggage of "this is instrumented in x and I can't change that" in addition to volume and cardinality considerations, perhaps it's a point-of-sale in a shop that only connects to the internet once a day for upload of the data).

But for the simplest solution, you're probably talking Prometheus + Loki/Elastic + Tempo + Pyroscope/Polar Signals + Grafana. This is simple as in "all pieces are proven and work in the small and get you fairly far"... but you still have to compose the solution out of those parts. And later you may wish to graduate to Grafana Cloud when the pain of running it all yourself is taking more time and effort than running the systems you're supposed to be working on. Oh, and if you really want to lean in on traces for everything, take look at Honeycomb too.

As to "the monitoring costs more than the app"... yup, at Cloudflare for a very very long time there was little monitoring of DNS requests or billing of it... because the act of counting it and recording it cost more than just serving the request. This principle scales well (or badly if you're looking at your credit card bill)... and is also why those who make these systems really have to keep in mind how to make the instrumentation cost less (eBPF?) as well as the ingestion and retention cost less (various methods, including aggregation, sampling, etc).

> the cost of monitoring an application can easily exceed the cost of hosting the application even for simple applications

(especially if you are looking at saas monitoring for a single-box or serverless thing)

I suspect SIEM is the most established use case for ingesting logs to something actionable; wonder if there are concepts there which port generally.

Datomic's approach to change capture seemed very good when it came out -- I wish this were a thing I could cheaply turn on and off on SQL columns. As things are, every ORM seems to resist having a table with no primary key so 'log important things to the DB' is hard in practice.

In postgres, mvcc makes it expensive to edit a scalar, so it is impractical to store counts. (There is an old uber blogpost I think about why they use mysql for this use case).

Specifically for alerting, I wish I could define application logic statistics in my application rather than PromQL / datadog json. Would be much easier to express 'warn me if this ever fails', 'expect 50% failure', 'warn me if too many retries' type stuff.

On the 'looking at logs' side, the hosted monitoring tools I've used are often uncomfortably slow, even when my data is small. I understand it's a hard problem but I miss graphite / whisper.

(not even getting into mixpanel / posthog type product-level observability)

1: https://opentelemetry.io/docs/concepts/signals/traces/#span-... A span event is data that is associated with a point in time rather than an interval.

I think there's two primary reasons:

1. Historically it was too hard to get easy value early. That's probably changed with OpenTelemetry since it has so much automatic instrumentation you can install. Even 3 years ago it just wasn't feasible to install an agent or write 5 lines of code and then get most of the libraries you use instrumented for you. Now it is, and it's fully portable across tools.

2. Cost, which still isn't solved yet. Tail-based sampling is still too hard, but you need it if you want to do stuff like always capture 100% of error traces but get only a statistically representative sample of 200 OK traces. There's some solutions for that today (e.g., my employer Honeycomb has such a tool), but it's still locked away with vendors and too hard to use.

I do expect the narrative on tracing to change in the coming years, though. OpenTelemetry is growing in popularity and it has enough features and stability for a wide range of people to adopt it. I think the cost issue will also be dealt with over time.

This is one of the reasons I strongly dislike the idea of remote or telepresence inspections of factories. You need to just walk the factory floor and use all of your senses (including smell). If you do this enough, you'll know when something is wrong. The pitch of the machines, the vibrations, the hum ... and you'll catch it before your rules do.

A personal example:

rsync.net has almost zero fraud detection for payments and signups. However, I, personally, have been watching the real-time order flow[1] for decades and I just know. A fraudulent signup just sticks out like a flashing red light. I could not teach this to you and I could not codify my heuristics.

I have spent decades thinking about how to augment - or replace - these real-time human observations with warnings and alerts and thresholds and I am deeply skeptical of their value relative to human interaction with the functioning system.

[1] ... with tail ...

[1] https://github.com/SigNoz/signoz [2] https://github.com/uptrace/uptrace

DEBUG - anything which would otherwise be a 1 line code comment

INFO - all business process relevant actions. Anytime something is committed to a database is a fairly good place to start.

WARN - abnormal outcomes which collectively might indicate a problem. i.e. a warning log should be printed when something succeeds but takes much longer then it should, or if I'm going to fail an action which will be retried later. Anything which logs at this level should be suppressible by reconfiguring the application to define the behavior as normal.

ERROR - Anything which should work, didn't, and isn't recoverable. Actions requested by the user which don't succeed should log at this level, since we're failing to meet a user request.

But now you can ask ChatGPT GPT4 to author PromQL and Jaeger queries for you, which it is incredibly good at.

The real, actual pain point - not cost, not maintenance, etc. - between the bottom line affecting stuff like customer success management and metrics was always turning the English language problem into the thing that goes into the computer. And that is seriously going away, the only people who think it's not are the experienced ones who, for some reason, months into the distribution of this insane technology, haven't tried asking for a PromQL query yet!

In my opinion, the way to avoid many of the problems and complexities that author lists is to start with the goals. What are the business/mission goals? What is the workload (not the monitoring tool) trying to accomplish? What does the thing do?

Once you have that, you can go about selecting the right tool for the job. The author notes that there's no consensus on what logs are for. Or is buried under signal-to-noise ratio for metrics. But when you start with a goal, and then determine a KPI you want to obtain to measure that, you'll then be able to ascertain where a log, metric, trace, or combination thereof can reflect that. And you only need create the ones needed to measure that. The scope of the problem becomes far more manageable.

It make take a couple tries to get right, and a user may find that they need to add more over time. But the monitoring doesn't need to change often unless the deliverables or architecture of the workload does. And creating those should be part of the development process.

By starting with goals and working from there, monitoring becomes a manageable task. The myriad of other logs and metrics created can be safely stored and kept quiet until they're needed to troubleshoot or review an incident. That falls into observability.

Observability is something you have. Monitoring is something you do.

I haven't addressed the difficulties the author states in creating various traceability mechanisms, how to parse all of these logs from various platforms, or some of the other technical challenges he states. There are usually solutions for these (usually...), some easier than others. But by narrowing the scope down to what's needed to achieve your given insights, the problem set becomes only what's required to achieve that aim, and not trying to build for every conceivable scenario.

We are a 3 man team and we are holding around ~4m active series with around a year of downsampled data in a storage bucket. Doesn't really cost us anything other than some servers on a cluster. We also use Loki for logs and Tempo for tracing in the same pattern.

One of the reasons I like Kubernetes, it's rigid enough compared to the wild west of VM land, and utilizing operators and tooling can get you solutions that just work provided you have enough core knowledge to fill in blanks.

These are the key insights we learned running hundreds of services at Amazon scale and you'll learn something whether you're just starting, you think you know it all or you need some actionable ways of improving your operational excellence.

That's why Graphite is fading off - it just can't keep up with the growth of data volume.

Prometheus is a great tool, but now it is not enough of one server. So you are either looking for workarounds with federation, or running more complex monitoring systems like Thanos, Cortex, VictoriaMetrics.

> Maybe you start with an ELK stack, but running Elasticsearch is actually a giant pain in the ass.

And that's the last he says of it. But I've seen ELK stacks in organizations from 30 employees / 5,000 users all the way to 5,000 employees / millions of users and it scales well. I'm sure it continues to scale above that.

Or for Prometheus scaling options,

> Adopt a hierarchical federation which is one Prometheus server scraping higher-level metrics from another server. [...] The complexity jump here cannot be overstated. You go from store everything and let god figure it out to needing to understand both what metrics matter and which ones matter less, how to do aggregations inside of Prometheus and you need to add out-of-band monitoring for all these new services. I've done it, it's doable but it is a pain in the ass.

It's not a pain, it's like an afternoon of work. I don't see why you need to be deciding which metrics matter - collect all of them. And if you don't know how to do aggregations in Prom, learn! You'll need to know this to use it anyway.

Better to tell the marketing teams to set up their own monitors and metrics.

It could be Google Analytics, it could be UptimeRobot. Or whatever they want to use.

Good for them to set monitors of every page that gets more than 1% traffic.

Good to monitor for the HTTP Response Code (especially if there's a WAF involved), simple content check (like the page title, to make sure something didn't get redirected), "too many hops test" for more than 2 hops to get to a destination, page load speed, and of course uptime. Most tools also will alert you if your SSL Cert or Domain is about to expire. Kinda useful. Good to monitor on every major vanity URL redirect as well... so like mysite.com/some-promo.

Setting this stuff up is SO EASY... but few people do it correctly / consistently -- so it's good to ask them to check once a month if they have all the monitors they need. Just grab the GA data and set a monitor on any page URL that gets more than 1% traffic... easy to just drop the data from GA into a CSV and import it into UptimeRobot. 2 minutes a month and you have amazing monitoring. Really important for monitoring to be run from a 3rd party location -- not from the same server, or behind the same firewall as your corporate network, but run from a location similar to what your real users are seeing.

From the dev side... having good logs so you can diagnose issues. But Let's face it, nobody has time to monitor and respond to things unless it's been escalated. So just let the marketing team / ecommerce team escalate the things they care about, and just make sure you've got enough logs and ways to diagnose the issues they report.

If you monitor CPU or RAM usage, you are doing it wrong. Oh and by the way, you want your ram to be full.

These problems resonate with me for sure, and after tackling them at other companies, they were part of our team’s inspiration for building Heii On-Call [1] as a lightweight, minimal monitoring platform for apps / APIs / Cron Jobs / etc.

Instead of “ingest all the metrics/logs” we have guides showing, for example, how to set up the most minimal possible Prometheus / Alertmanager, with the smallest footprint of CPU/memory/disk requirements. [2]

And for many of our users they don’t even use any of our Prometheus or Datadog integrations, and just stick with our dead simple out-of-the-box HTTP monitoring. [3]

Note that these address the “now” problem of monitoring: is it working now, or not? While most of the article talks about the “past” problem of monitoring (long term storage of logs, metrics; pretty charts).

The “past” part is the part that the author is pointing to as being rarely useful and the source of the cost and complexity. I think there’s probably still value in dumping whatever logs and metrics you want into S3 just in case you need to go digging one day, but for the most part, this is pretty low value. The relatively higher value and lower implementation complexity is in the “now”.

[1] https://heiioncall.com/

[2] https://heiioncall.com/guides/minimalistic-monitoring-and-al...

[3] https://heiioncall.com/docs

And I can monitor a whole k8s cluster (around 70 nodes) using 100MB of mem and 3G for storage (up to 1 year).

Keep it simple!!

The logs (including text logging, telemetry) are just signals.

The way to improve monitoring is by continuing to refine s/n ratio & critical alert health through continuous review of signals and identifying leading indicators.

Every week you should identify the critical objectives and the leading indicators that represent healthy and unhealthy states for those. Also identify false positives and noisy alerts and work to reduce noise.

For example, how many people are checking out, adding to cart, viewing the website. What are indicators of health and error conditions. What are true signals and what are noise?

The biggest issue i see is people setting up logging and alerting without continuing to revise and improve the process.

Logging is at the application level. Log just enough that you can replicate a problem, usually it means just the input.

The rest is marketing.

Our primary mission around monitoring is to proactively respond to customers before they call us regarding exceptions in their environments. Az Monitor w/ basic alerts are all we really need to get that job done.

After spending 20 minutes with these tools, I felt really dumb for rolling our in house solution the last ~8 years. All we had to do was take a look and try. There is absolutely no way in hell that my team can compete with the resources of a trillion dollar hyperscaler like this.

The long game I am looking towards is the partnership between MS & OpenAI. "AIOps" is already listed as a "Respond" option under Az Monitor in their documentation. Anyone who has been standing on principle against Microsoft has to at least see some of the light coming through this tunnel by now, train or no train.

"Log Levels Don't Mean Anything"

If you're not handling log levels correctly, you've just thrown away half the utility of logging. The author doesn't even make an argument here, he just lists a bunch of different log levels and moves on. You should absolutely rely on logging levels in your applications.

"No consensus on what logs are even for"

They're for an application to tell you what is going on. Come on dude.

"Maybe I'm wrong about monitoring."

You're not wrong (well, about being a pain), you just should get a different job. Monitoring is hard, yes. But it's very beneficial, despite being highly imperfect. It just doesn't benefit you to build your own monitoring stack. Pay someone else to do it, accept that it's not going to be the Platonic ideal of computer science theory, and move on with life.

- we are all doing monitoring wrong? No, sorry.

- You set up these tools with the mentality of "set and forget" but they actually require ever increasing amounts of maintenance? The tools and your monitoring rules require maintainance like everything in IT.

- Logs OK, don't use logs for monitoring.

- Metrics Yes, do use time series for monitoring.

- Prometheus is really designed to be running on one server. You can scale vertically ... Actually you can operate more than one Prometheis behind a load balancer, avoiding the situation "when there is a Prometheus problem you lose visibility into your entire stack at once", and without needing to federate them.

SaaS observability gives you the tools to control these costs. It's much, much cheaper to log up to debug if you're not indexing them most of the time. If you don't understand features like exclusion rules, you should go learn them.

There is one common function to format as low-level HTML, and a trivial markup code to simplify calling it.

Just because log utilization rates are often very low, doesn't meant that the .0001% isn't immensely valuable.

[1] https://grafana.com/oss/mimir

Levels are easy. You have three, DEBUG, INFO, and ERROR. DEBUG is stuff that's interesting to you, the author of the code. INFO is things that are interesting to the person/team that runs your software in production. ERROR is for things that require human attention.

My opinion is that INFO logs are the primary user interface for your server-side software, and demand the level of attention that you would expect a frontend designer to devote to your brand or whatever. That's where the software gets to interface with the person that's responsible for its care and feeding. Garbage in there wastes their time. Everything in INFO should be logged for a reason. (Yes, sometimes there's spam. Usually request/response data. Most of the time the requests go great, but you don't know they've gone great until you've logged the fact that they were made. It's a necessary evil.)

Meanwhile, DEBUG should probably be pretty chatty. If your program is consuming resources like CPU, it better justify itself in the DEBUG logs. A program sitting there using 100% of the CPU and not logging anything at level DEBUG is very very suspicious. My philosophy is to "show your work" on every critical operation. Log when something starts, what decisions it makes along the way, and when it's done. A message like "failed to contact server example.com: i/o timeout after 10 minutes" is 10 minutes too late to be interesting. "connect to example.com: started", "connect to example.com: i/o timeout after 10 minutes" is much better, because 10 minutes before the error you can read the logs and realize example.com is NOT where your database is hosted. (You can really go overboard here and I would never stop you. connect to example.com: resolve DNS: start, connect to example.com: resolve DNS: A 1.2.3.4, connect to example.com: dial 1.2.3.4: start, .... In production, I compromise and log "start", "ongoing long HTTP request" (at 10 seconds), and "finished" with the status code.) Every branch is a place to log which branch you selected and why. If there's a bug and you only know the outcome, but not the software's "thought process" that led you there, you have nothing to go on. "Hey, the FrobTheBaz endpoint is returning 400s." That is useless information. You have to read the code and guess. You do not want to be guessing during an outage.

Errors are a whole can of worms. If someone reading the logs can't fix the problem, it's not an error. "Error: cannot write data to the database". That's a big problem. Disconnect that popcorn popper and plug the power cable to your database back in! "Error: login for root from spambot@1.2.3.4: invalid signature" is a problem for your firewall. What is someone reading the logs going to do about that? Call up 1.2.3.4 and ask them to stop guessing SSH keys? At best, that's an INFO log. There are 3.4 with 38 zeros possible keys. They aren't ever going to guess it. If you decide "this isn't even worth logging", I can definitely agree with you. (I'd log it, though. An increase in failure attempts is good ancillary information when something else draws your attention to the system.)

Finally, newline-delimited JSON ("jsonl") is the only correct format for logs. Nobody wants to write a parser to figure out "how many 400s did we serve today". Just let JQ [https://jqlang.github.io/jq/] do it for you. If you want pretty colors for viewing, use something like jlog: https://github.com/jrockway/json-logs. You don't need to write VT100 control characters to your log file. I know for a fact you don't even have a VT100.

All other formats of telemetry are just logs with fancier user interfaces. Metrics? Just write them out to your logs: "meter:http_bytes_sent delta:16384 x-request-id:813db394-75a1-4b22-94ad-e7850d9cdd25". You still have the ability to tail your logs and convert the metrics to a proper time series database (with less cardinality than "per request" of course), but you can now also do investigations in context. For example, I've implemented exactly this system at work. A customer shows up, saying that we're using too much memory. I ask for the logs and see something like "span start: compaction", "meter:self_rss_bytes delta:2000000000", "span end: compaction, error:nil", "meter:self_rss_bytes delta:-1000000000". Hey, we leaked a gig of RAM in compaction. You would never figure out the memory leak with "compaction done after 5 minutes" and a graph of memory usage across your fleet.

While you weren't looking I also threw traces in there! Have a logging primitive like Span("operate(foo)", func { ... }) that logs the start and the end, and throw a generated ID in every message. Concatenate the span IDs which each recursive call to Span (I abuse context.Context for this, sorry Rob Pike), and log them with every message in the span. Now you have distributed tracing. As always, sure, parse those logs and throw the log-less spans in to Jaeger so you can efficiently search for rare events and have a nice data flow graph or whatever.

Log sare so simple and so great. The problem you run into is by not producing enough of them. Disks cost nothing. You can delete the debug logs after a short period of time, or not even produce them until you suspect an issue. Don't sweat the log levels; pick the 3 I picked and try to enforce the rules in code review. Aim the log messages at the right audience; dev team, ops team, emergencies. Show your work. Log contextual information. Put metrics in the logs, and aggregate them for dashboards and monitoring. Annotate logs with trace IDs, and put those into Jaeger. You'll never have to wonder why your system is broken again; you printed out everything it's doing. The aggregation will tell you you're not meeting your error budget, and the logs for a representative request will tell you why that one failed. Once you know what's broken, the fix is always easy.

Whenever I tell people my system, they tell me it can't scale. When I worked at Google, I wrote exactly this system for monitoring Google Fiber (well, we didn't have log levels really...), and we could answer questions about the entire CPE fleet as easily as we could answer questions about one device or one bug. Over many years, I've finally gotten this into the "real world" (like weeks ago), and it's been amazing.

Good logs are the best thing you can do for someone that has to run your software. For a lot of folks, that's you! For other folks, that's your customers. Can you customer set up Prometheus, ElasticSearch, and Jaeger, and dump that all out and send it to you in Slack? Unlikely. Can they dump a file that's 10MB of JSON lines into Slack? They sure can. And now you can diagnose the bug and fix it without bothering them for more info. It's good for everyone.

I lived this for years and there’s just no easy solutions here.

I don’t care what kind of application you’re building, if you want to be able to reliably deliver good service and iterate in any type of scientific fashion, you must-have great observability. This is not a suggestion. It is simply creating the situation where you can run experiments and understand the impact of changes to your system.

But that means you need to significantly invest in not only the technical tooling, but the people, education and procedural things that are necessary to think in a data oriented way. That is something that is excessively hard to convince people to do because it’s so seemingly boring.

The real siren here is push button observability, and automation like with solar winds. Everyone would love to be able to have State machine level determinism, and monitoring to tell you which distinct state you’re in. I’ve yet to find an org that can comfortably manage and exploit that level of granular observability.

This is an amazing article that every engineer should read.