Statistical process control after W. Edwards Deming (2020) | Hacker News Reader

Statistical process control after W. Edwards Deming (2020)

2uo.de

81 pointsTomte47 comments

W. Edwards Deming is the source of one of my favorite quotes: "In God we trust; all others must bring data."

One of the fascinating things about Deming's work is that Japan's automobiles went from being referred to pejoratively as "Jap Crap" to symbols of quality (Toyota, Honda, Subaru, Nissan, Mazda, etc.) within a short few decades--quite a feat for a nation struggling to regain its industrial footing after WWII.

Deming used the quote, but did not coin it: https://quoteinvestigator.com/2017/12/29/god-data/

I'm curious what makes that your favorite quote.

It gives me pause, mainly because I've seen it abused, ironically, often in contrast to some of Deming's core ideas.

YMMV

My first job out of college (undergraduate) was working for a small consulting firm that did a lot of work in this area for the petrochemical industry and their suppliers. The owner had a PhD in decision sciences (statistics) and he did consulting work on statistical process control (SPC). This was my first exposure to this field and was so fascinated by it that after working there for a year, I quit my job so that I could go back to school full time and pursue a Masters degree in this area, but with focus on management information systems (MIS). Ironically, my work after graduate school has never touched on SPC or quality control.

It's unfortunate that Deming (and many others) never really got the attention that their methods deserved.

Twenty-five or thirty years ago, one hear a great deal about total quality management (TQM).

Yep, it was back in the early 1990s.

I'm hoping that will change. I'm a big fan of Deming and Drucker, they've both got really insightful thoughts on business.

My goal is to work more of Deming into knowledge work (software) to develop meaningful improvement into the process and product while leveraging Drucker for better effectiveness of the individual.

GE and other companies tried to implement Demings approach. They called it Six Sigma. Those that used Deming's concepts were successful. Yet too many misunderstood the idea of SPC and used in areas that was not meant for quality control. For instance trying to use SPC in the managerial process itself.

Gawd, Six Sigma was an ever-present factor in all our work when I was at IBM in the late 80's to early 90's. I have complete respect for Deming, but when anything becomes "religion" it gets damaged or mis-used a lot, as you say.

In our software group, we were constantly battling with management that wanted to start to look at software development as a factory-like process, and to apply factory QA processes.

Cargo culting one’s way to attempt obtaining results seems like the default approach of leadership unable to really adapt the proper lessons from other organizations and apply them to their own. If I as a person look up to a world class bodybuilder as a scrawny weakling, the solution to achieve that goal isn’t to just eat like a bodybuilder but to train like one and that usually means dropping other activities like playing a bunch of video games (not that it’s mutually exclusive - see Henry Cavill). But we as humans do the same thing all the time (how many people think of rich people in terms of consumption or some sets of personalities from popular media rather than to look at the data fully). However, there is a lot to learn and enjoy in the process, and companies that do well learn to have clear goals and processes that seem genuine and that employees can connect with. When people stop feeling this connection, they simply stop doing the habits at the individual (not working out, relapsing, etc) or disengage in organizations (burn out, disillusionment, etc)

Six Sigma is to the field of Statistical Process Control what PowerPoint templates are to the field of graphic design. The biggest problem with Six Sigma is many of the people who roll it out understand how to follow the instructions in a Six Sigma book, but don’t understand the underlying statistics well enough to recognize when the templates will fail or need to be changed.

Amen. The principles behind Six Sigma are solid. They're just a repackaging of Deming's SPC principles after all. However, too often it devolves into a box ticking exercise. That being said, in my experience just educating everyone in your organization on the basic principles (e.g. what Six Sigma would call yellow belt) tends to be quite valuable. Even without a lot of statistics, teaching people to first properly define a problem statement and emphasizing finding root causes before taking action tends to lead to better decision making.

Six Sigma is a travesty of statistical process control. For one thing, only rarely does a process attain six sigma reliability. Six Sigma is about cult-like in-group/out-group dynamics for the Six Sigma Black Belts.

I’m reading a book called Working Backwards about Amazon. There is a chapter on metrics and it says to use the DMAIC process, which comes from six sigma, for developing reporting and metrics. The authors provide an example about applying this to e-commerce at Amazon. They don’t seem to use control charts, but they are still trying to examine the metrics for common vs special cause of variation.

If you can relate inputs to the outputs of a process and you can get reliable data on the inputs, then you can use SPC in most areas. The costs to do so many not be worth it, however.

Which managerial processes do you think this doesn’t apply to?

Amazon, the Supply Chain part of it at least, has one of the best metrics systems I have ever seen in my life. Every metric is measuring something meaningful, each one is measuring a crucial process interface, each one is based on a simple calculation. Also most are measured in DPMO, looking at defects instead of things meeting expectations. The funny thing is, during my time metrics where presented, across all levels, in blunt, no chart, tables in a auto created PDF. And yet this is still light years ahead of everything else I saw, including fancy control towers.

Until recently, there hasn’t been enough data to apply statistical process control in a lot of fields. Chemical process companies (like Kodak used to be) have used SPC. “Web scale” companies commonly talk about running experiments now. Lots of people talk about A/B testing, which is an extremely simplistic example of SPC. I’ve been anticipating more advanced SPC tools to replace A/B testing for years, but am not aware of any. Maybe I should have created one.

Now I wonder if Machine Learning might pass up SPC as the primary method by which people identify and control key process variables.

Actually, it does not take a huge amount of data to make use of statistical process control. In fact, it's so little that someone should be able to do it with paper, pencil, and calculator.

The statistical technique I think that's better able to displace A/B testing is design of experiments (DOE). This approach also does not require that much data to implement. There was an excellent youtube series on it where the presenter used popping popcorn for his experiments. I can't seem to find it now.

I've observed DOE in use. It's a statistical method at its core, and as such potentially suffers from the replication crisis.

I stumbled upon Deming's "Out of the Crisis" when wandering through the library in school. It has changed how I thought about system improvement ever since.

I use the things I learned for marketing, product, and engineering systems.

Want to improve a system? Before doing anything, collect data and measure. A system must be stable, else you cannot improve it. Marketing campaigns. Funnel optimizations. Performance improvement.

Most "stability" is achieved by simple things like squashing bugs (campaign bugs, performance bugs, or UX bugs).

Once you achieve stability of output (i.e. low standard deviation), then improve the system.

Simple, uh?

Well, keep in mind that "improvement" here implies non-transformative change (we usually call this "optimization" on informatics). If you are not sure you have the correct system running, stabilizing it is the last thing you want.

I think that may be a too narrow vision of “transformative” improvement.

I worked in a factory where one of the biggest, and hardest, improvements we made was coaching the managers and engineers to leave the technicians alone when they were doing maintenance.

We collected data on how often they were interrupted, how much time it cost, and how interruptions could lead to RIT changes in procedures that affected quality.

Previously the engineers and managers saw the checking as helping because they could se ethe status of things and observe and answer questions, many would even lend a hand. Oversight and engagement was seem as a path to improved quality. To let go of that was a huge transformation. It started with data - a lot of data - that allowed us to identify that weekend and night maintenance generally went faster and better.

Well, I claim that is a way too broad definition of "transformative". The people kept creating the same products, by almost the same procedure.

> If you are not sure you have the correct system running, stabilizing it is the last thing you want.

Stable in this case just means you are clear what the system is and can keep operating it if you want to.

That is a prerequisite to evaluating the system in order to determine whether it is what you want.

>"You can not inspect quality into a product." ~Deming

Yep. This applies double to software. Putting software under load and trending health measures to look for two standard deviations variance is very similar to how Deming instrumented production lines. The neat thing that software enables for both automobile manufacturing and software products themselves is that those exact same health measures can be trended and analyzed after they get into the hands of consumers.

If your quality apparatus is purely inspection based, you will live your life one escalation to the next.

I've spent much of the past few years trying to work out if this means I should stop relying on pull requests to catch problems, and put something in place ahead of writing code instead, but so far I've not figured out an alternative that works. If anyone has any ideas, please share them.

One of the reasons why pushing change straight to production with A/B testing works so well is that you can expose your new change to statistically significant real user load and, assuming your have decent instrumentation, find the problems. This does mean that a subset of your users get exposed to those problems.

A more user friendly way would be to have a scaled down replica of production setup with simulators that mimic user behavior. Assuming the development team actually believes in and tends this environment, they will find the lion's share of bugs that commonly slip through peer review.

Pair-drafting pseudocode with a colleague into a Dropbox Paper document is a great way to evaluate tradeoffs.

Deming is talking about repeatable manufacturing processes. The point of Statistical Process Control is that you sample metrics that tell you if the process will produce products reliably within tolerances.

Software is completely different, it is hardly ever a repeatable process to create software. Making the same software over and over is nonsensical, but making a billion identical contact lenses is exactly what you want.

There are forms of software "inspection," like pair programming, that amount to what Deming says you should not do in manufacturing: Inspect everything. But pair programming is pretty rare. Other practices like code review have been called into question. So it may still be true that you can't inspect your way to quality in software, but not for the same reasons that it is a bad idea in many cases in manufacturing.

You are 100% right that measuring the process of creating software is a terrible application of statistical process control. See Jira.

One way to look at how Deming/Shewart's work applies to software is to consider what happens when you apply their statistical analysis techniques to running software.

> Software is completely different, it is hardly ever a repeatable process to create software. Making the same software over and over is nonsensical, but making a billion identical contact lenses is exactly what you want.

I've heard it said like this: Creating a software is like developing a recipe for a cake. Baking thousands of cakes (a "manufacturing process") is what you do once you have the recipe. The two activities are very different.

Kind of. We have to clarify what we mean by "creating a software".

This seems to be rarely done in most discussions on processes for software development. The lack of information regarding this means that everyone gets to be right and angry at everyone else.

Are you making novel systems? Then it's like developing a recipe for a cake. Are you making bog standard CRUD web apps? Then you're baking thousands of cakes.

A lot of software development sits somewhere in between these two extremes. It has repeatable, well-established portions that can be executed almost mechanically, and portions that require you to sit down and collaborate or really think about what you're doing. How these balance out on a project determines what approaches are appropriate to even consider.

Then there's software maintenance (an unfortunately neglected aspect of software in many parts of this industry). Here the same consideration applies. Are you writing new software that's really just generating new kinds of custom reports? That's rather mechanical, you may even be able to automate yourself out of a job. Or are you adding truly novel features, extending your document editor to support real-time networked collaboration?

Software runs the full gamut so no statement about what software is will ever be correct. We can only discuss the utility or applicability of processes once we specify which kinds of software activities we're involved in.

Certainly some software looks routine and feels repetitious while you are doing it. But very little if any software is amenable to determining its manufacturing tolerances. That phrase is jarring because the idea of manufacturing tolerances for software does not make sense. But manufacturing tolerances are fundamental to statistical process control, which then builds on statistically valid sampling to determine if part of a process is failing. (Or it can determine that a quality issue is something other than a process failure.) But it depends on setting and using manufacturing tolerances wisely.

Deming gets the credit, but in Japan it was Homer Sarasohn[1][2] who did the foundational work[3]. In fact, he handed the reins off to Deming.

One particularly interesting thing is that, at least once upon a time, Canon didn't even bother inspecting their copiers and printers, because the variance was so low it was a waste of time[4].

[1] https://en.wikipedia.org/wiki/Homer_Sarasohn

[2] https://honoringhomer.net/

[3] https://apnews.com/article/e2908339e16a48dc8efe37d77a905daa

[4] http://qualiticien.over-blog.com/article-679016.html

This is great; had never known about Homer Sarasohn before. So many amazing people have been lost to the vagaries of history that it behooves us to research and rediscover our "lost" heroes.

Thanks for all the links; can you recommend some introductory books on the subject?

On the subject of "lost" heroes? Not really, I've just come across a few in the course of my life. It would certainly make a great book though!

I was asking about books on Statistical Process/Quality Control though a book on our "lost heroes" would make a more enjoyable reading :-)

It seems Japan's resurgence after WWII owes a lot to the enlightened policies of a "few" Americans and their willingness to actually follow through with concrete actions. I wonder how many Japanese really know about their own history and what they really owe the US.

The irony is that Deming’s philosophy was much better received in Japan that in the United States. American cars were considered crappy during the 70s and 80s while the Japanese were iterating their quality practices. Deming was not the only philosopher-guru of manufacturing quality. There was Peter Drucker, Walter Shewhart (Shewhart cycle), Joseph Juran, Shigeo Shingo, Philip Crosby (quality is free, rework is costly), Taguchi (Taguchi methods), Ishikawa (fishbone). JIT, Kanban, Kaizen funny enough are Japanese manufacturing terms that have made its way into our programming dialect.

Also funny that the US defense industry was very close to some of these concepts during WW2. Only to have the Japanese adopt them really efficiently afterwards.

The funnel experiment examples are interesting. The fourth case (T' = H) is a great example of normalization of deviance [0]. Without a clearly communicated target to return to, you reliably shift further and further from that initial target, not even oscillating around the target like in the third case.

[0] https://en.wikipedia.org/wiki/Normalization_of_deviance

This is a great article, but is there a good modern source (say an introductory text book) on these ideas?

I've a vague memory of doing this as a first year undergraduate, but that was a long time ago. And stupidly, I didn't think I'd ever need them...

Deming wrote very accessible books. Out of the Crisis is a general audience book but it covers statistical quality analysis in enough depth to understand how it works.

Which parts of the concepts are you looking for?

Douglas Montgomery has a textbook called Statistical Process Control that covers control charts and many other statistical techniques.

I recommend "Understanding Variation: The Key to Managing Chaos" by Donald J. Wheeler, colleague of Deming. Good intro to SPC.

Thanks for the tips guys. I’ll check them out. I also pulled a copy of Box’s essays off the shelf, and while he is obviously mainly interested in experimental design he does drop in a few references to oldish books on SPC.

The products I design/sell generate the inputs for this kind of thing for the semiconductor industry.

It always makes cocktail party conversation complicated.

Out of curiosity, do you mean that your products generates template forms for collecting statistical sample data (like Minitab), or that your products collect measurements that are fed into tools like Minitab as data?

Here are some online resources on Deming's ideas

  https://curiouscat.com/management/deming/online

Reply on news.ycombinator.com