Beware of Tight Feedback Loops (2020)
brianlui.dog
brianlui.dog
"We did A, it was slow, then we did B, and it was fast." == "Congratulations, you discovered that the cache works, not that B is faster than A."
"We ran this test on AWS image size such-and-such, and it performed 5% better than the competing product that we tested previously." == "That's just noisy neighbours on a multi-tenant platform, you've learned nothing."
"We ran a test and it succeeded, hence it is now working!" == "The system fails nine out of ten times randomly, you just stopped when you had your first success. Try it again."
This ability to spin up a test environment quickly and run it through a scenario can be intoxicating. People try all sorts of random stuff, see a "win", and then call it a day without really doing the legwork to prove that they achieved something meaningful.
In practice, it's not critical to go that far. In my experience, huge performance wins are trivial to benchmark with just a modicum of effort, and tiny performance wins are typically not worth any effort at all. At FAANG scale they might be, but I don't work at a FAANG, so not my problem! :)
Simple methods I use and can recommend are:
- Ideally, collect data from production, where the change behaves exactly as it would in production because it is production!
- Interleave tests on the same machine. E.g.: A, B, A, B, A, B, etc...
- Run tests on every machine you can get your hands on over an extended period and draw a histogram or a scatter chart of some sort. You don't have to do this at 100% load, it's actually better to mix this in with normal workloads. This gives you an idea of the noise versus signal, platform variance, variation due to time of day, etc...
- Run tests after a cold boot to get a sense of worst-case performance instead of idealised and often unrealistic "hot loop" performance.
- Run tests with different input sizes and draw a graph. Try find the explanation for every feature of the graph.
- Be aware of every cache layer in the system. Don't forget about the potentially huge RAM or SSD cache of the storage subsystem. Typical SAN arrays now have 256 GB or more. The previous graph above can help, but only if your input size goes waaaay past the biggest cache sizes in the system.
As a real-world example: I recently reduced the size of a database platform by a few vCPUs to save on licensing costs, but increased the per-core performance and simultaneously did some network tuning. Some benchmarks showed the same performance, others showed an improvement. Both were extremely noisy before and after the change.
I would like to take credit and say that I improved performance while saving money. However, the scientifically honest result is that any performance change is statistically insignificant.
The cost saving is significant, because it has very little variance. That's good enough, and I don't have to make a statement about performance that the numbers can't really back up.
So many load tools send a request then wait for the reply then send the next. That is basically useless for understanding variance under spiky load and for assessing when a system is below its capacity how stable it is.
For real world performance, the variance matters a lot. The code will make some things be cached some not (so the data variability in the test load should follow the same sort of power law that your real traffic does), and even if you get 10% faster at the median if the variance goes up the quality is lower.
But you don't need to teach people all the fundamentals of calculus before they can apply it. Definitions and a small set of pre-made answers are enough to start using it and focusing on the statistics. When (and if) those people decide to learn it more in depth, they can go back to calculus fundamentals, and will have an easier time doing so, because they will know why those things exist.
If yes, then I agree with him. People should have more probability lessons at school than trigonometry or calculus.
It only becomes useful as you move from simple combinatorics to continuous distributions, at which point you need differentiation and integration to switch between CDF and PDF. Much in the same way trigonometry becomes useful in physics once you need to deal with vectors and waves.
Trigonometry in particular is so ingrained in many aspect of applied science that not learning its fundamentals and then pushing to its higher level concepts is going to seriously reduce adaptation capability in students.
And if you somehow never need to rotate things, it's a good training in applying theoretical rules to concrete problems. In my university, after 2 years of increasingly complex trigonometry, we were left with 3D engine programming and to the fact that opened my mind forever that theoretical mathematics have a purpose.
E.g. you would always choose to read an accelerometer sensor in a tight loop in a flight control system - however, you wouldn’t just blindly apply the latest reading as an input to your system (accelerometers are extremely noisy, the data looks like complete trash on the front end before applying some kind of averaging filter).
Further down the chain on a flight control computer is also a good counterpoint to this idea that you should prefer a more lossy sample rate.
Control theory is an established field of study. Better to adopt filtering & PID control strategies than to under sample an input.
Sounds a little unconvincing.
> applying some kind of averaging filter
How do you choose the size of your averaging window?
How do you make sure you don’t catch a systematic bias due to some unrelated event?
Control theory may help in some cases (flight control) but possibly less so in others (stock market).
Yeah, it is pretty wooly. How about taking it from the opposite position, almost like a proof by contradiction, e.g. the opposite position might be “you can see better if you look less often”.
In that example you can have excessively tight feedback loops where looking less often let’s you see better.
The article is talking about "feedback loops" in some handwavy business-marketing-whatever bullshit advice sense.
This paragraph sums up the motivation of the article quite nicely:
"But after achieving proficiency in a field, tight feedback loops are useless. That’s because initially, the learning environment is gentle. The path is well-travelled, there are easily accessible guides, things work according to common sense. As we become proficient, the environment becomes harsher, with noisier feedback. Improving is not as easy. At higher levels of skill, further progression depends on self-learning, on discovering or inventing new practices and knowledge. As skill increases, the gap between optimizing for metrics and optimizing for mastery widens."
Obviously noise is a problem. That's a separate issue from loop tautness. Tight is a measure of loop size not accuracy of information as the article claims. This is similar to accuracy vs precision.
It's worth adding that whether or not the right thing is measured is not related to feedback loop size. The "timely" 737 Max example that was peppered in shows this. Clearly what was being measured was not related to the problems the plane had.
The unsubstantiated (not even anecdotal) claims based on the author's confusion (e.g. data-less diagram of masters preferring loose loops) seem to be the author taking his own advice. They're complete speculation, not even pretending to be attached to reality.
The technical details of failures in MCAS system weren't the problem. The problem was the decision chain that caused creation of MCAS system, and that one had nothing to do with tight or loose control loops, and everything to do with business incentives perversing safety process.
Software development is a highly people-driven environment. You can introduce something in your process (e. g. a new framework) but the feedback comes only a year later when it becomes evident that the framework hasn’t been a good fit or was introduced poorly (e. g. team members are quitting, productivity has plummeted, tech debt has piled up).
In the time it takes most professions to consider trying a new framework or tool (beyond the early adopters), entire programming languages rise to popularity and fall back into obscurity.
I heard the same sentiment from an architect friend. (The kind of architect who designs buildings) He said that in their business you put your best ideas on paper. Most plans then don't get built for reasons outside of your control. Projects fail, companies fold. Sometimes your designs can't pass review with the authorities because you misunderstood the regulations. Sometimes it can't pass review because the local government doesn't like your customer. Years down the road some of your plans are built. If there are problems with the design some problems might be obvious immediately, but most will only show up as the building ages. And if there are problems, are they there because you designed it wrong? Or the builders skimped on materials? Or is it maybe because the costumer meddled with your designs? You will never know. The feedback loop is both years long, and the execution is inaccurate.
On the other hand besides being an architect he was also dabbling in hobby programming projects. There if he made a syntax error the computer pointed it out to him within seconds. And even when he found a bug after compilation he could be certain it wasn't because the compiler saved money by using 10% less screws than specified or something. That is to say the feedback was both fast and accurate. If you had bugs, most probably it was because you made a mistake.
I also recognise the fuzzier process of designing features with stakeholders, implementing them, deploying them and having them tested in the market as something between the Rocket Launch (for the technical process from ideation to deployment and subsequent maintenance) and Stock Picking (for the product/market fit).
To finish populating the 2x2 matrix, the Novice Bowler approach is more similar to what many of the students used to do back when I was tutoring Algorithms and Data Structures at uni. They'd throw things at the IDE until it compiled, and call it a day.
This is a really important point that the author sort of glosses over -- adding measurements, or tighter measurements must be done extremely carefully for two reasons: 1) the risk is 'looking for the key under the streetlamp because it'd better light', and 2) unintended side effects.
The first will cause data to just be basically irrelevant or even inverse to the desired signal, but people will chase it anyway because it is easy to measure. The second taps into the management phenomena of "you get the results that you measure", and create perverse incentives to do the wrong thing, either self-managing, learning, or managing an org.
Good article, much more worthwhile than I expected!
I'm actually thinking that this is part of what is going on with all the cries of "Inflation!!" - compared to last years' numbers, when everything was running at about 40%, of course prices are rebounding, and even overcompensating on the high side - the real question is for how long (there was also a massive hole blown in the money supply, which needs to be backfilled).
┌────┬────────────────┬──────────────────┐
│ │ Accurate │ Inaccurate │
├────┼────────────────┼──────────────────┤
│ │ │ │
│ │ │ │
│Fast│ Math test <-- Novice bowler │
│ │ │ │
│ │ ^ │ ^ │
├────┼───────|────────┼────────|─────────┤
│ │ | │ | │
│ │ │ │
│Slow│ Rocket launch <-- Stock picking │
│ │ │ │
│ │ │ │
└────┴────────────────┴──────────────────┘The novice bowler example has nothing to do the tightness of the feedback loop. The issue is solely the statistical variation in the results.
The novice bowler sometimes bowls too far to the left, sometimes to the right. This isn’t a result of conscious action, rather a lack of control. When the bowler aims for the middle of the lane and the ball randomly goes to the left on one occasion, aiming to the right on the next shot will not fix the problem; it will make it worse.
Summarizing the real issue for the bowler: don’t make decisions based on random data. If you don’t know to what extent the data you’re looking at contains a random element, you should try to figure out how much before attempting a correction.
Lengthening the feedback loop in this example might involve obscuring the bowler’s vision of the pins once the ball has begun rolling and presenting a video of the results 10 minutes later. It’s hard for me to imagine how this could be anything other than detrimental.
Back to the bowling example (sorry for harping on it), unless you frequently hallucinate and see the pins fall when they did not, the results in practice are very very accurate. It’s just that they describe a complex system with an initially unknown amount of variation. It’s only through having precise measurements that you can even quantify the variation.
I think I was introduced to the concept in some talk about AI, something which I never really pursued. Nevertheless, the concept stuck with me and I've been reminded of it professionally more than once.
"Next, we eliminate some incorrect worlds by testing them against noisy data. This data should come from intermediate steps and can’t be quantified. If the data conflicts with an imagined future, that future is unlikely to occur. Noisy information may be bad for feedback loops, but it’s great for elimination, because the wider range of outcomes is more effective for pruning. It may even be a bit faster than accurate data, since the variance of accurate data is lower!
As we near decision time, we stop the world construction and elimination, and use judgment and experience to pick the best idea out of the remaining ones. Process improvement happens by creating many parallel processes and discarding the ones that are unfit, not by iteratively improving a single process."
I can't quite understand what he is saying.
Essentially, you identify numerous hypothetical scenarios, analyse the impact they would have, and use that information to decide whether a particular course of action is a good idea or not.
This creates a faster feedback loop inside of the larger, noisier one. Your feedback is now guided around the question of "what makes the rubric itself better?" This can be done on principle, with limited access to external information. Philosophical thinking is eminently suited to this style of problem, but it can be supplemented with short-term empirical studies that add some falsifying points and narrow your cone of uncertainty.
At the end you've generated a list of yes/no questions forming the rubric of whether the course of action is likely to succeed. It can be turned into a ranking score, or a pass-fail threshold.
If you're frustrated by the idea of just making it up on principle, that's a frustration with philosophy itself; it rarely "works" until you accept some pragmatic premises around what is "good" or "true". The point of having a large number of questions, using a wide variety of perspectives, is that they test the overall coherency of the premise. Something can work fine from one perspective, and then completely fail in another. When that happens, it's a good sign that you have more to improve.
It's quite an important life skill to practice. It's easy to go along with the crowd, but this is a way of breaking away from it.
i feel like we are conflating tight feedback loops with some other concept that needs to be teased out.
small bets can be divergent.
I get it - we live in a money-oriented society.
But presenting a psychological learning model as useful as this one and limiting it to the stock market feels ... narrow minded. It sounds like "being a jedi is nice because you get this fancy tool with which you can cut AND braise sausages at the same time".
Use this to get more healthy. Use this to spend more and better time with your family and friends. Use this to live a more fulfilling life. And sure, use it for the stock market or your day job. But that could be a couple phrases at the end, not the "meat" of the thing.