Suppose I wanted to kill a lot of pilots

https://en.wikipedia.org/wiki/Pre-mortem

I didn't used to take this seriously. Then one project I was on was delayed by several weeks when members of the team got jury duty, got injured, and had other medical emergencies in a rather short period.

We still got it done, but I got a lesson that month in planning.

I've been a part of a few large scale system implementations and we documented every significant risk to the project at the beginning and as we went if new ones presented themselves. This seemed to be a standard part of the PM's playbook, and I used to to save critical organizational capabilities when things collapsed:

--------------------------

I raised risk on a must-not-fail deadline, but it actually failed and was the camel straw that brought the project down. (An Oracle product) PM's were didn't appreciate the risk. There's still probably an Oracle VP out there who really doesn't like me for using the established project protocols that allowed me to escalate the issue to the president where I worked and way above the VP in Oracle. On my own initiative I kept the legacy system-- supposedly partly decommissioned-- updated just in case so we could fail-over softly. This was unexpected: When failure became obvious I reminded the VP that I had raised the original risk, and then I had prepared for it. I think their plan had been to box us in to meet their increased $ demands.

It was part of another missed deadline required for a major milestone 25% payment on the contract, and it was a fixed-price contract so they should have absorbed the costs.

Instead, when the failure progressed to my area, we all showed up to work one day and everything was gone. No contactors, nothing was left, like a cleaning crew had cleared it out. A week of negotiations failed when they wouldn't budge from demands for extra $$millions to keep working on the project, plus their missed 25% milestone payment.

Ultimately I lost a years worth of work when things entered lawsuit land and sued each other for breach of contract. (we won-- or settled at least. The fixed price contract was clear, and the missed deadline were partly due to demos of fake functionality that didn't actually exist or components they knew would be EOL'ed after signing the contract & before implementation would begin. The case for outright fraud was pretty strong: We had videos of the demo.)

In case you're wondering, I don't like Oracle. Though with a different product we were still forced to use Oracle DB's. Those I actually don't mind and were a huge step up from the ~2000 MS SQL Server & much older DEC VMS file-based database that wasn't SQL compliant or relational.

If anything happens due to one of those foreseen issues, everybody knew about it in advance and already discussed what, if anything, could have been done to prevent it as well as what action was actually taken.

I love the SAFe / PI Planning approach because it makes sure that everybody is on the same page and totally removes any blame game from deliverables. Far, far fewer surprises.

Imagine it's 6 months from now, and the project we're about to begin is half complete and twice over budget. What factors will the post-mortem identify as causes for the failure?

Learn the lessons before the project begins, not after it fails.

once you hace committed to play the estimation game, you're committed to the delusion and reality checks dont matter. Just stop trying to estimate.

https://asq.org/quality-resources/fmea

>put all state in one place.

>no caching.

Just as a counterpoint, both as a user and a developer I've found the main cause of performance issues and UI bugs come from a combination of distributed state and caching.

State being stored as a private member on the widget object itself is arguably more likely to cause a state desync bug or accidental O(N^2) behaviour than everything being global. IMGUI solves this but has only really seen a warm reception amongst the gaming industry (where, incidentally, UIs are always lightning fast despite being traditionally limited by older console/handheld hardware.)

Having a cache at the wrong level might reduce some function's execution time from 100us per frame to 10us, but makes it much more likely that the entire app will become unresponsive for 10s while some dumb bit of code rebuilds the whole cache 25,000 times in a row.

In a similar vein, I've found issues in multi-threaded apps where some function is sped up from 10ms to 2.5ms by making use of all four cores, but occasionally spins on a mutex, blocking the UI for 250ms if some slow path (usually network/disk IO) is taken.

Genuinely, I think the simplest way to make sure a program will have bad performance just requires one step:

- Make it difficult for a human programmer to reason intuitively about the performance of the code they write.

> - concentrate the app in as few machines as possible

Well, sorta. Distributing an application across multiple machines only reduces downtime if the machines are independent of each other.

I've seen bad architectures where wider distribution leads to more outages, not fewer. I even consulted for a company who had their services in two data centers for redundancy, but they put different services in each one, so a failure in either data center would effectively bring down their entire stack.

It may feel like that for somebody with little experience with FP, but half of those points are design decisions that aren't changed by the language paradigm, and most of the rest is a non-sequitur where the stated problems aren't caused by the characteristic that precedes them.

Immutability makes chaching easier, not harder, the same for propagating by reference. Besides, there's nothing on FP that asks for more IO operations or distributing your software. The item about larger data structures is on point, although the indirection is very often optimized away on practice, while imperative programmers often have to create their own indirection by hand, that the compilers have a harder time to put away.

Anyway, most of both lists apply perfectly to the modern "mandated microservices" architecture some place use, that I think was your point.

No, not really. For example:

> - make it hard/impossible to set the true state of the system in its entirety

Strangely enough, environments like Smalltalk allow you to do exactly that, but other environments not so much.

That sounds like a very exact description of React/Redux.

Say that you are implementing a role-based access control system.

And then ask yourself how can my RBAC system fail?

Well one way it could fail is if someone without the necessary role can operate on something that they ought not have access to.

So then create roles A, B, and C, and create a resource and say that group A can rw, and group B can r this resource. Then create a user and give that user role C. And then try to use this user in that role and try to read the resource. If it can, fail the test. Then another test where you try to write using such a user. If the resource was modified, fail the test.

This is as opposed to only thinking about what you want to happen, in which case you might be writing tests that only ensure that those that should be able to read/write can do so. And of course you want to test that too. But ensuring that those that should not be able to access a resource cannot is the more important thing to be sure of, and also the type of thing that might slip by unnoticed.

A system where those that should be able to access resources cannot, this will be detected in normal operation anyway, by way of users performing their usual tasks. But accessing resources you should not is the most critical and could go by unnoticed for a long time.

And that is why the tests that are written in the backwards thinking fashion are the most important ones.

Basically, at the project planning stage, I get everyone together and ask how this shit is going to blown up in our faces. What are all the scenarios that are complete failures.

Then a few weeks before a release, I do the same thing. It's amazing the issues this catches from the cross team approach. For instance, something that the product manager was worried about, they never brought up because they assumed engineering knew about it was covered, but in reality, they had no idea that was important.

Or with two engineering teams, or DevOps etc. Normally we have more than a dozen action items out of these meetings.

I think of it less of reverse the problem and more changing my frame of reference or perspective of the problem.

Also similar in nature is safety engineering. You think about how to make the system fail easily and unexpectedly and you avoid those.

You also try to prioritize the highest risk/lowest cost issues in these. These all involve risk management, which is also critical in investing and is a major reason why Munger has been so successful - he only goes for low risk/high reward plays and bets big on the few opportunities that he gets.

I started with what actions people take that make the bathroom as disgusting (pee everywhere, vomit, shoving food in the walls, etc) and then determining defensive mechanisms from there.

- Does X ALWAYS cause Y? - Can Y happen WITHOUT X? - Could X cause OTHER things that cause Y?

The point was not tho explain HOW to avoid planes crashing, it was to explain how he saw his job not that of providing super-clear weather reports ,but focusing the reports on factors that would cause problems for pilots.

I think that's the essence of the article.

[1] https://www.cracked.com/article_18839_7-planes-perfectly-des...

So for the dust filter, maybe selecting a material that's only rated for high temperatures in short bursts. Or one that needs frequent, labor-intensive replacements in that dangerous environment. Or test that your filter survives at high temperatures, and test the filtration efficacy, but forget to test the efficacy at high temperatures.

Using the other example - of course to kill the most pilots you coukd out them in a big meat grinder or just blow up their planes. But the point is to design planes so that they both fly and kill pilots.

It's funny because HN (in contrast to the world at large) seems to be frequented by a lot of people who are gifted with the forward-looking contingency mindset. I'll bet that's why you got upvoted, and it's probably why the article was posted.

One reason why a community like this would exist online is because it's effectively a saf...err I mean friendly place for people who are concerned about future events, but in a positive, problem-solving sort of way. The rest of the world doesn't always reward this or want to hear about it.

IT, operations, and even software development need people who can think this way (particularly the opportunistic side of software dev, as it is more COTS-components oriented and thus less NIH and thus theoretically more battle-tested/hardened against future events).

Sadly one of the things I've noticed about those with this mindset is that if they are not able to act on their perceptions constructively, for example if they were raised in an environment where it was considered a crazy way to think about things (focus only on past or present), it can turn into a more subjective, fear-based practice. In those cases it seems you get more outcomes like a stance where even just going outside feels to a person more like gambling with their life. Just observationally speaking...

Repeat until you run out of ways of losing money.

Just think about it. I think the best way to lose a lot of money is to fall into scams. And avoiding scams is an important part of investing. It also tell you that scams can be profitable. So the next question is: how to scam and lose money doing it, which will hopefully put you back on the right track... or make you a really good scammer. Back to honest investing, losing a lot of money is not that trivial. It is easy to take risks but you can still get the occasional big payoff and if you think about ways of not getting that payoff, it is the same as thinking about how to get it, just with a different angle.

That kind of reverse thinking goes best when you pair it with regular thinking. That's a "meet in the middle" algorithm, kind of like solving a maze by going both forwards from the entrance and backwards from the exit.

If you start to remove the roads to failure you're kind of forced onto the road to success - or at least a place where you're not falling into the places you're planning around.

Buy great companies cheap or at a good price. Only sell if you think they are technologically threatened (horse and cart vs car situation) or have grown to take up too much of your portfolio (like up 50%+ of your portfolio).

This is basically how buffet got mega rich. Amex, Coca Cola, Apple, Geico, etc. These few big bets made him a very rich man and he still holds them today.

Losing 1% per year to transaction/currency/ management fees per year really adds up over a lifetime. And it doubly matters when you are rich enough to pass the cap gains threshold. 20% lost on every time you sell at a profit in the UK - that compounds too!

Or - if you can't be bothered with the effort to try and beat the market - you've also illuminated another bit of buffets advice. Invest in and hold an s&p ETF through your whole life.

In fact Munger is one of the biggest proponents of "invert, always invert"

I mean you did already start to think of some interesting starting points:

> But the way to do that is just to churn my book a lot and pay costs and spreads.

Don't churn your books, reduce costs, reduce spreads

Iterate the question. Assuming a limited number of trades per quarter and a cap on spreads and commissions, how would one maximally lose money?

Well it suggests to invest long-term?

If you have strategy that very quickly loses money in this harsh conditions, you have a winner.

I suspect the second feature is more important than the first. Feedback from real failure remains more valuable than feedback from imagined scenarios. Thing is, real failure can also be quite expensive.