Spc-kit: A toolkit for statistical process control using SQL
github.com
github.com
I am one of the folks that for dashboarding prefer to push everything to SQL VIEWS and functions. So I use this to monitor proportions for different processes month to month, which may have error bars +/- five to ten percent.
It is technically not the right test to know if the process changed, but is useful by eye to know typical variation.
I use the binomial CIs in dashboarding scenarios, so if the user selects a different subset of data, all the graphs are auto-updated (on the SQL side). Which when you get people subset into tinier slices of data it becomes more important.
So if you can upfront identify the processes you want to monitor, then when flagged have this send an alert. That is better than a dashboard forcing people to click on stuff and hope they identify a anomaly in the process.
The background is that I've thought about applying SPC to software behavior for a long time[0]. While I was at Shopify I began to think about using it for detecting regressions in the YJIT benchmark suite[1]. My first thought was to code a Ruby library (Shopify is a Ruby on Rails outfit), but I decided that it would be more accessible to more folks if it worked inside a database instead. I got caught in a 20% layoff at Shopify and that somewhat took the wind out of my sails, which is why it's been dormant for a year.
There are some limitations to be aware of in a software context.
First, sample sizes have to be constant. In a world of unreliable systems and networks this is a pretty tough constraint. It could be achieved by taking fixed subsamples at random from a varying larger sample (eg, normally we get 100, pick 50 at random). Switching to variable sample size would be possible, but it will be a lot of work and I haven't had the motivation to tackle it yet. It's also only lightly treated in the main textbook I've worked from, because the usual focus is manufacturing operations where fixed sample sizes are common.
Second, it doesn't deal with non-parametric distributions. Classical SPC is rooted in the Normal distribution, and continues to work quite well with any distribution that has some kind of centrally-located mass that tapers off to the edges. But a lot of software behavior follows power laws, especially Pareto distributions. There is SPC literature to deal with non-normal distributions using non-parametric statistics. But I haven't bought and digested the relevant books.
It's worth noting that SPC is as much a system of problem solving as it is a bunch of statistical tools. Most major metrics vendors now offer some kind of anomaly detection, but then what? What are your out-of-control action plans? What is your concept of an acceptable amount of staying close to the mean? What is your target, and why is it your target? Can you even achieve such a target with the system as-is, or are you dreaming? SPC attends to these questions as well.
I can definitely recommend kqr's introduction[2], which is linked from the repo.
[0] https://theoryof.predictable.software/articles/what-is-predi...
[2] https://two-wrongs.com/statistical-process-control-a-practit...
I'd also be interested in applying this to software for better metrics, just thinking about how I might be able to get somewhere quickly in an innovation/hackathon days (aside from your project) came across https://grafana.com/grafana/plugins/kensobi-spc-panel/ - looks like it doesn't actually do any calculation, you have to feed it your measurements & sub-group sample size etc. but could be handy in conjunction with yours.
That Grafana plugin looks awesome. If I can shave off some time from endless leetcode grinding I might play with it.
I can't think of a reference for the different charts off the top of my head. Certainly working SPC-kit helped me to better understand the differences and how they relate to each other. But that's not a very scalable way to transmit information.
Just a quick question though if I may - why/when would I want a n>1 group to average instead of treating them all independently as in XmR? Is it if you have multiple categories (of say machine operators, production lines, or software servers) and that variable (the category) isn't the one (x) you're interested in? So for example the first sample is the time taken by each of 8 machinists to fabricate the first nail, etc., so each x_i bar is the mean time for a nail over time (i). Or if we were interested in comparing the machinists, then x_j would be employee #j and for each one we'd be taking the mean of their n nails.
Two reasons come to mind. First, for whatever reason, it might not be economical to measure every individual in the population. There might be too many, or the thing being sampled is a continuous flow (eg liquid product in a chemical processing plant) or perhaps the sampling is destructive (imagine a pressure failure test, or a reagent test).
Second, you might just naturally have a sample. For example, each YJIT benchmark is run multiple times because there's variability in the measurement not due to changes in YJIT. Since you have n runs of the benchmark, you naturally have a sample size of n. It doesn't make sense to think of them as consecutive measurements.
The rest of your comment goes to the business of selecting what the grouping of samples is (the term is "rational subgrouping"). A lot of pages are given to this question in the SPC books I've read, because it's not difficult to unintentionally mask signal by combining things that shouldn't be combined. An example is checking the precision of piston machining. It might be you take all the pistons in a given engine as your sample. But later you discover that one of your four machining stations is off-center, which was masked by the ordinary variation of the other three. In this case the sampling should have been per-station, not per-engine.
Do you happen to know which they are? This has intrigued me for a long time but when I asked Wheeler he was skeptical of doing it that way.
In the past I've also tried the trick of taking the logarithm of values to convert a power distribution into a more normal-ish distribution. My project was to see if I could alert on an increased rate of uploads to RubyGems, signaling a possible malware campaign. Time between uploads follows a power law. I used the logarithm trick to convert it -- though not successfully. There are actually multiple distributions -- the "random" one which is prominent, but also other peaks centered around 1 second, 2 seconds, 3 seconds etc. Basically the detection was made difficult because of mass uploads being rate-limited on the uploader's side.
My current simplistic (and very dumb!) solution that I've used for power-law type distributions — like HN virality, for instance — is to count the number of days between viral events, and then subject that to process control.[1] I basically take Wheeler's approach to chunky data and use that for J-curve type data, which tells me if the behaviour of my 'HN virality process' has changed.
I'd be very interested to learn of other approaches.
[1] HN traffic for commoncog.com displays routine variation most weeks with an Upper Process Limit of 192 and a Lower Process Limit of 0, unless one of my articles hit the front page, at which point I get 11-16k additional uniques).
I did forget to bring up the Poisson approximation you mention though. I'll include that too.
One possible way out is to look for measurements that contribute to running time but which are not affected by other factors. I remember the YJIT folks talking about using CPU instruction counters, but I can't find it on the benchmark website.
Unfortunately the GitHub Linguist grammar for SQL doesn't recognize PostgreSQL comments, so highlighting will be wonky in places.