Observations on Observability
blog.colinbreck.com
blog.colinbreck.com
I wanted to write such a book, or half of such a book, as a second part of Knative in Action. I wrote a simulator[0] and did a ton of reading which felt like I'd barely scratched the surface of the surface.
But really, there's just no room for what I wanted to do. Based on a simple estimate it would have run to about 200 pages on top of another 250 pages of actually-about-Knative material. So that whole section wound on the cutting room floor. Maybe another time.
In terms of books I felt helped me the most down this road, here's a reasonable reading list I'd point to:
Feedback Control for Computer Systems: Introducing Control Theory to Enterprise Programmers, by Philipp Janert. Short, to-the-point introduction to basic classical control theory using PID controllers.
Matching Supply with Demand: An Introduction to Operations Management by Cachon and Terweisch. A short, meant-for-MBAs textbook on operations management (ie applied operations research). Simple and approachable. Very good for warming up to...
Factory Physics by Hopp and Spearman. Magisterial in scope, also magisterial in grumpiness and idiosyncracy. I've directly applied material from this book and Matching to research work.
Business Dynamics by Sterman. Still the best system dynamics book I've ever read. I'm anxiously waiting for the 2nd edition, expected circa 2021.
Performance Modeling and Design of Computer Systems: Queueing Theory in Action by Harchol-Balter. Approachable and enlightening. I stumbled on a few of the proofs and got lost once or twice in thickets of notation, but that's due to my own mathematical immaturity more than the book itself.
But I still don't have a book to refer to for the specifics of software dynamics, spanning its physics and economics. And I wish I did. I would love to read that book, if folks can make recommendations.
> Fortunately, as discrete pieces aggregate, they start to look continuous. At scale, we can take advantage of this, embracing higher-level metrics that can be treated as continuous signals.
He's saying that rather than look at request-level metrics in your Big Data Message Queue (request ID, etc.), your message queue in aggregate can be seen as an industrial process, and there's an entire discipline built around managing those called Statistical Process Control.
Seen in this way, the next NewRelic is going to be all about creating p- and c-charts [1] where the limits algorithmically turn knobs on your kubernetes cluster.
In other words, the next big dev-ops trend is going to be the return of Six Sigma.
I think that actually, observability has always been one of the weakest areas in tech -- we reinvent a lot of it ourselves while the wider world abounds in rich concepts of measurement, inference and control. SPC, control theory, inference engines, classifier systems, measurement theory, design of experiments, fuzzy sets, Dempster-Shafer theory, avionics, system dynamics, cybernetics and frankly a bajillion other disciplines that I've never heard of and never will.
Sometimes in R&D, the R is much more profitable than the D.
Which point, I should say, the article makes reasonably well.